I set RAG abstention thresholds by the cost of a wrong answer - not confidence alone. If an error costs £5,000 and an unnecessary refusal costs £2, stricter checks may make sense. Those figures are illustrative; your workflow needs its own cost estimates.
I use three checks, each with a different job:
- Retrieval relevance: Do the passages match the question? A high score does not prove they contain the answer.
- Evidence coverage: Does the source text support every material claim? High-impact claims must pass this check.
- Model uncertainty: Does the answer remain stable, or does doubt call for another check?
I compare these checks alone and together on labelled questions, then test the chosen thresholds on held-out data. I track errors among released answers, the share of requests answered, false refusals, review load, cost and latency.
The goal is acceptable risk at an acceptable cost. That means choosing when to answer, qualify, retry or abstain - while keeping safety and privacy restrictions above any score. A confident answer must never cancel failed support.
::: @figure 
{RAG Abstention Thresholds: From Evidence to Action}
:::
1. Retrieval-score thresholds
Answer support
Retrieval scores help route a query; they don’t decide whether to abstain. BM25 matches terms, dense similarity matches embeddings and rerankers reorder candidates. All three score relevance, not answer support. Even a high-scoring passage may leave out a required date, exception or condition.[5][7][8]
Failure modes
These limits affect whether a gate should open at all.
| Signal | Interpretability | Calibration burden | Typical failure mode |
|---|---|---|---|
| BM25 | Explained by term matches and document-length bias | High across queries and indexes | Rewards keyword overlap but misses paraphrases |
| Dense similarity | Measures semantic proximity, not support probability | High across models and document types | Finds related passages that don’t contain the answer |
| Reranker | Provides useful relative ordering of candidates | High across models, domains and candidate sets | Puts insufficient evidence first; cannot recover missing documents |
| Normalised or fused score | Easier to combine | Still needs deployment-specific testing | Can create false confidence |
Reciprocal Rank Fusion combines ranks without needing comparable raw scores. Its output still reflects ranking preference, not answer correctness.[4][5][6]
Calibration needs
Label passages for direct answer support, not topical relevance. Test a range of cut-offs on development data, then measure unsupported-answer rate, answer coverage and false refusals on held-out questions.
Revalidate after changes to chunking, models or the corpus: scores may not transfer across queries, document types or indexes. Higher top-score thresholds reduce unsupported answers but increase false refusals.[3][9][10]
Cost and role
Use retrieval scores as an early routing check, not the final gate. Keep first-stage retrieval broad enough to find supporting passages, then rerank a shortlist. Reranking adds processing cost and latency as candidate counts and passage lengths grow.[2][4][7]
When retrieval is strong enough to proceed, the next check is whether the evidence covers what the answer needs.
2. Evidence-coverage checks
Answer support
Once retrieval passes, check whether the retrieved text supports each claim. Retrieval scores select candidate passages; evidence coverage tests whether those passages support the answer.
Extract material factual claims, breaking compound statements into single claims where needed. Compare each claim with the retrieved evidence and assign one of five statuses: fully supporting, partial, irrelevant, absent or conflicting.
Measure coverage across material claims that need citations - not words or documents: coverage = fully supported material claims ÷ total material claims.[12][14]
Failure modes
A citation counts only if its passage entails the claim. Check exact numbers, dates and scope. Keep contradictory evidence rather than picking the source that agrees with the draft.[12][13]
| Check | Support assessed | Limitations | Cost |
|---|---|---|---|
| Claim-level coverage | Whether each material claim has enough evidence | Missed claims inflate coverage; claim extraction must be reliable | Extraction, evidence matching and aggregation |
| Citation entailment | Whether the cited passage entails the attached claim | Can miss unsupported qualifiers or multi-step reasoning | Claim–passage comparisons using rules or a verifier |
| Cross-source consistency | Whether independent sources agree or conflict | Shared errors, stale information, dates, negation and differences in scope complicate assessment | Source clustering, cross-source comparison, temporal checks and possible escalation |
Calibration needs
Use labelled claim–evidence pairs covering all five classes, including compound claims, stale documents and duplicated sources. Human reviewers should resolve difficult cases.[11]
Set thresholds by claim importance, not one global score. Every high-impact claim needs full support. Supported background facts must not offset an unsupported legal, financial or safety claim.
Cost and role
Run a fast first pass to extract claims, check for citations and spot obvious contradictions. Reserve deeper entailment checks and source comparisons for high-risk or borderline claims.
Measure cost and latency per answer. For audit, retain claim text, evidence identifiers, source versions, verdicts and the evaluator’s rationale.
State fully supported claims directly. Narrow or qualify partial claims, and drop irrelevant or absent claims. For unresolved conflicts, retry retrieval or abstain unless a source-authority rule resolves them. Keep this gate separate from model-uncertainty checks.
This gate filters out unsupported claims; the next check asks whether the model remains uncertain despite the evidence.
3. Model-uncertainty thresholds
Answer support
After retrieval and evidence checks pass, uncertainty signals help decide whether an answer is stable enough to release. A confident answer is not necessarily a supported answer. Use uncertainty as a second gate for borderline cases, not a replacement for support checks.[17][21]
| Signal | Signal quality | Relative cost | Susceptibility to overconfidence |
|---|---|---|---|
| Model confidence | Token or sequence probabilities show hesitation during generation, not necessarily factual accuracy | Low when token probabilities are available | High: fluent but unsupported claims can score well |
| Verifier confidence | Confidence from evidence checks helps route answers; reliability depends on domain testing | Low if reused; moderate to high if rerun | Verifiers can confidently approve plausible but unsupported claims |
| Answer-variation uncertainty | Differences in meaning across sampled answers show instability or ambiguity | Moderate to high: requires multiple generations and comparisons | Repeated answers can share the same error |
| Context sensitivity | Comparing answers with and without retrieved context shows sensitivity to evidence | Moderate: requires another generation and comparison | Misleading context can increase confidence without improving correctness |
Signal quality depends on calibration and operating conditions. No signal provides a universal acceptance threshold.[17][22]
Failure modes
Compare sampled answers by meaning, not wording. Differences in numbers, entities or conclusions matter more than phrasing. Track changes with and without context separately, and treat unresolved disagreement between model answers as instability.[18][19]
Calibration needs
Map uncertainty signals to labelled correctness and support outcomes for each question type and risk level. Include cases that are unanswerable, ambiguous, outdated or based on conflicting evidence. Test reliability on held-out data, and set stricter thresholds when wrong answers cost more than refusals. Recalibrate after changes to the model, prompt, corpus or verifier.[16][17][22]
Cost and role
Use token-confidence features for low-cost initial routing. Reserve multiple samples or comparisons with and without context for borderline or high-impact questions.
Use uncertainty to choose whether to answer, qualify, retry or abstain. Uncertainty should supplement retrieval and evidence gates, never override failed support. Log signal values, routing decisions, latency, cost and eventual outcomes.[17][20]
Should Your RAG Answer or Abstain? Context Sufficiency Classifier + Full Code Walkthrough
Comparing single-signal and combined gates
The choice is not which signal to use alone, but which combination controls risk at an acceptable cost.
Signal strengths and limits
Each signal has a different job: retrieval filters candidates, evidence coverage checks support, and uncertainty flags remaining risk. Relevance does not prove support.[23]
| Signal | Principal strength | Main failure mode | Calibration cost | Role in pipeline |
|---|---|---|---|---|
| Retrieval score | Fast, inexpensive filtering | High scores can still miss answer support; low scores can miss paraphrases | Label answer support; retest after index or model changes | Triage and retry trigger |
| Evidence coverage | Checks claim support | Can miss valid synthesis or multi-document support; evaluators can also err | Label claim support; test evaluator precision and recall | Primary grounding gate |
| Model uncertainty | Flags ambiguity or competing answers | Confidence can be miscalibrated and sensitive to prompt or context order | Calibrate on held-out questions; monitor risk–coverage | Warning and escalation input |
This division of labour works only when the gates run in sequence, rather than acting as interchangeable checks.
Single-signal, all-pass and risk-scored gates
A retrieve → evaluate → gate → act pipeline turns abstention thresholds into an enforceable release rule. Retrieve passages, check relevance, claim support, contradictions and coverage, then apply the gate before release. This can block or rewrite a fluent but unsupported response.
| Gate | Decision rule | Benefit | Limitation |
|---|---|---|---|
| Single-signal | One selected score passes its threshold | Simple, low-cost baseline | Leaves other failure modes unchecked |
| All-pass | Retrieval ((r \geq t_r)), evidence coverage ((e \geq t_e)) and uncertainty ((u \leq t_u)) all pass | Explicit conditions; easy to audit | One noisy signal can cause an unnecessary refusal |
| Risk-scored | Combine calibrated features into estimated error probability or expected loss | Accounts for signal interactions and business costs | Needs representative labels and regular monitoring |
An all-pass gate uses fixed pass conditions. A risk-scored gate estimates risk from the signals together. Neither should let a favourable score cancel a hard evidence or safety failure.
Stricter gates mean fewer unsupported answers, but more refusals, retries and reviews. Check this trade-off with a risk–coverage curve: coverage is the share answered; selective risk is the error rate among answers.[15][25] Choose thresholds using observed error and refusal rates, not by maximising confidence alone.[24]
When to answer, qualify, retry or abstain
Use the combined gate outcomes to route each response:
| Signal outcome | Action | Pipeline enforcement |
|---|---|---|
| Adequate retrieval, high evidence coverage, low uncertainty | Answer | Release supported claims after policy checks |
| Partial coverage; acceptable risk for an independently useful subset | Qualify | Remove unsupported detail, cite retained claims and name the gaps |
| Weak retrieval; no override | Retry or clarify | Reformulate the query, broaden retrieval or request missing detail |
| Missing evidence, unresolved conflict or persistent high uncertainty | Abstain or escalate | Block unsupported output and identify the evidence needed |
| Safety, privacy or regulatory restriction, regardless of scores | Refuse or escalate | Apply the restriction before ordinary routing |
Apply overrides before retry or release rules. Weak retrieval must not trigger further searches for prohibited information. If sources conflict, report the disagreement only when it is supported, safe to disclose and allowed by policy. Otherwise, abstain.
Before use, fit these release rules to labelled questions and the cost of each error.
Calibrating thresholds with labelled questions and error costs
Thresholds work only when you fit them to labelled questions and test them against the cost of each mistake. To turn the gate into a tested policy, label examples, measure outcomes and choose the threshold that minimises expected loss.
Build and split a labelled dataset
Label questions as answerable, unanswerable or conflicting against the permitted corpus. Use the evidence gate’s five support labels. Score answers for correctness, grounding and completeness against the context shown to the model, and label retrieval, support and uncertainty outcomes for the same questions.
Log retrieval score, coverage features, uncertainty, action, latency, token use and cost. Include policy look-ups, multi-document comparisons, stale-document conflicts and absent-corpus requests.[26][27][1] Use this same labelled set to compare retrieval, evidence and uncertainty gates on identical questions.
Freeze the index, prompt, model and scoring rules before splitting the data. Keep same-source, same-version and near-duplicate questions in the same split. Fit on development data, calibrate on validation data, then test the locked policy once on a hold-out set.
Keep difficult cases - especially conflicting and unanswerable questions - in every split, and report their proportions. Report workflow results with bootstrap intervals, rather than a single point estimate.
Measure answer coverage, risk and running costs
Define final outcomes before testing: direct answer, qualified answer, retry-and-answer, refusal and human escalation. Count qualified answers only when they meet the release rule. Count retries within the original request, and score escalations separately.
Test each single-signal and combined policy across thresholds at matched coverage or matched risk. Include uncertainty intervals and use comparable review budgets. For calibration, state the event being predicted: factual correctness and support for every material claim need separate labels.
| Metric | Definition and denominator | Decision question |
|---|---|---|
| Answer coverage | Answered requests ÷ all requests; count qualified answers only if they satisfy the release rule | How much demand is served autonomously? |
| Error rate among answers | Incorrect or materially incomplete answered requests ÷ answered requests | How risky are released answers? |
| Abstention rate | Refusals ÷ all requests; also report false refusals ÷ answerable requests | Are useful answers being withheld? |
| Unsupported-answer rate | Answers with one or more unsupported material claims ÷ answered requests | Does release require enough evidence? |
| Claim-level evidence coverage | Supported material claims ÷ all assessed material claims | How fully is each answer grounded? |
| Selective risk at fixed coverage | Error rate among answered requests at the stated coverage | Which policy is safer at equal service levels? |
| Calibration error | Gap between predicted and observed event frequency across score bins; report bin sizes | Does confidence mean what the gate assumes? |
| Escalation rate | Requests routed to human review ÷ all requests | Is review capacity sufficient? |
| Cost and latency per request | Total operating cost ÷ all requests; end-to-end p50 and p95 latency across requests | Are checking and routing affordable and timely? |
With these metrics defined, translate the results into business loss.
Choose thresholds by expected business loss
Use per-request probabilities, not error rates among answered requests, in the cost calculation. Define final outcomes for every request, then calculate:
[ \begin{aligned} \text{Expected loss} ={}& P(\text{wrong answer}) \times C_{\text{wrong}} \ &+ P(\text{refusal}) \times C_{\text{refusal}} \ &+ P(\text{escalation}) \times C_{\text{escalation}} \ &+ \text{operating costs}. \end{aligned} ]
Account separately for retrieval, verification, retry, delay, compute and missed-opportunity costs. Do not count any cost twice. If outcomes overlap, use an explicit additive cost model.
Estimate probabilities per request over the same population. Include the cost of falsely refusing an answerable question, not just the cost of saying no. Choose thresholds that minimise expected loss while meeting limits on unsafe answers, minimum service coverage and available reviewer capacity. Choose the point where lower unsafe-answer risk justifies the lost coverage.
Workflow consequence Illustrative wrong-answer cost Illustrative refusal cost Illustrative escalation cost Gate strictness Low: internal research £20 £8 £15 Lower to medium; allow some residual risk where users can verify Medium: customer-support policy guidance £250 £20 £35 Medium to high; require strong claim support High: clinical, financial or safety-critical use £10,000 £150 £80 High; require stronger evidence and escalation for unresolved cases
Replace these hypothetical figures with your organisation’s incident losses, staff time and customer-impact estimates. Test plausible cost ranges rather than relying on one estimate, and select thresholds that minimise expected loss within safety, coverage and reviewer-capacity limits.
Lock and test the chosen policy. Recalibrate after material changes to the corpus, model, prompts or request mix: a single 0.8 confidence cut-off cannot capture these differences.[28][29]
Conclusion: combine signals and calibrate for cost
Use retrieval for initial triage, then verify answers through evidence and conflict checks. Use uncertainty only to resolve borderline cases.
Calibrate the combined gate on labelled questions. Set thresholds using measured business costs and safety limits, aiming to minimise business loss rather than focusing on refusals or answer rate alone.
In production, track unsupported answers, refusals, review load and cost per request. After material changes, revalidate against a versioned regression suite. Tighten or roll back the gate if severe errors increase.
FAQs
How much labelled data do I need to calibrate RAG thresholds?
Start with 20 to 50 representative tasks, covering clear successes, failed runs and complex edge cases. Use this sample to set initial RAG abstention thresholds that meet your quality and cost requirements.
Next, test those thresholds on a much larger evaluation dataset. Validate the chosen policy on a separate, independent holdout set, then monitor and adjust the thresholds to keep quality at the required level.
How do I estimate the cost of a wrong answer?
For each use case, assess what a wrong answer could cost. This might include financial loss, legal liability or reputational damage. Assign a monetary value to each consequence.
Hokstad Consulting recommends calculating AI agent cost by dividing total delivery spend by the number of accepted tasks, including retries, tools and human review in that spend. Human-in-the-loop checks for high-impact decisions can help put a price on failures that need correcting.
What should I do when abstention signals disagree?
Do not rely on model confidence scores. Review raw telemetry, logs or metrics manually to identify the cause. If the evidence is incomplete or conflicting, clearly attribute each position to its source and explain any differences in scope, date or version. For high-impact business decisions, refer the conflict for human review.
Hokstad Consulting helps organisations refine these processes through AI strategy and implementation, supporting dependable, accurate information management.