RAG Context Compression: Token Savings vs Lost Detail

I’d choose RAG compression by answer quality and total cost - not token savings alone. Start with trimming when exact wording matters. Test summaries when passages repeat, but keep every claim linked to its source.

Here’s how I’d compare the options:

I’d also test a hybrid: trim first, then summarise what remains.

Before deployment, I’d check three things:

• Net savings: count every model call, report costs in £, and measure median and P95 response times.
• Answer quality: test facts, figures, exceptions and citation support separately, using contexts of 2,000, 8,000 and 20,000 tokens where model limits allow.
• Pass/fail limits: require zero critical unsupported claims and full citation coverage for legal or safety-critical claims. If an uncompressed baseline cannot run, mark savings against it as not measured.

My rule: keep the evidence your answer needs. If compression removes a required condition, restore the source text or abstain - even if the shorter prompt costs less.

::: @figure {RAG Context Compression: Quality Before Token Savings} :::

Contextual Compression in RAG Explained: Less Noise, Better Context

Compare extractive trimming and generated summaries

Test both methods on the same query set and report results by workload, rather than as one average. Use summaries when evidence overlaps heavily, and trimming when exact wording and traceability matter.[3] The test is not which method compresses more, but which preserves enough evidence for the next generation step.

The table below compares the main trade-offs for grounded generation.

Extractive trimming: Keep relevant source passages

Rank passages by relevance, remove near-duplicates and select evidence within a token budget. Support selection with lexical matching, embeddings and reranking.

Keep each fact with its qualifications. Expand passage boundaries to retain units, dates, definitions, exceptions and uncertainty. Keep table headings and footnotes when figures depend on them. Also reserve space for context that relevance scoring might miss.[3]

Preserve document IDs and sentence or character offsets so readers can trace retained text to its source. If a separate compression model selects spans, count its processing time and token usage in the comparison.[3]

Generated summaries: Shorten text and preserve source links

Use a query-focused summary to bring repeated evidence across chunks together without merging distinct entities or time periods. Check that it retains entities, numbers, dates, units, conditions, exceptions and uncertainty. Set a maximum length and a minimum evidence threshold: a shorter summary is no help if it drops a required qualification.[3][5][6]

Shortening evidence can break the link between a claim and its source. Store each claim with its source ID and passage offset, and keep identifiers stable through chunking and reranking. Treat the summary as derived text. Discard unsupported claims or send them for review before answer generation.[3][4]

With claims, offsets and source IDs preserved, measure whether the full pipeline still saves tokens and time.

Measure token reduction and net pipeline savings

Compare the two methods under the same conditions. Keep the query set, retrieved documents, retrieval settings, model, prompt and citation format fixed. Change only compression. Run the uncompressed baseline only if the full retrieved context fits the model window. Otherwise, mark it not runnable - don’t silently truncate it.[10][12]

For each workload and document-length band, compare:

• Original and final context tokens.
• Total pipeline tokens and estimated cost (£).
• Median and P95 end-to-end latency.
• Answer-output tokens, answer quality and citation support.

Mark unavailable metrics as not measured.

Count tokens and costs across all model calls

Measure the whole pipeline, not just the final answer call. Record original and compressed retrieved-context tokens, prompt tokens with and without context, answer-output tokens and total tokens for every model call. Track the compressor’s input and output separately from the final answer model, and include any selection or reranking calls. Use each model’s tokeniser rather than a rough word-to-token estimate. Report usage both per model and in total.[7][8]

Calculate each call’s cost as (input tokens × input rate + output tokens × output rate) / 1,000,000. Use rates per million tokens and account for cached tokens separately where applicable. Add the costs of every call and any separately charged processing.[11]

Net saving = baseline pipeline cost − compressed pipeline cost. To get percentage savings, divide the net saving by the baseline cost and multiply by 100. Calculate costs in the billing currency first, then convert to £. State the exchange-rate source, date and conversion assumption. If the baseline cannot run, mark its net saving as not measured rather than extrapolating a result.[11]

Measure latency across workloads

Test exact-figure lookups, conditional policy questions and cross-document synthesis separately. Group tests by original context length: 2,000, 8,000 and 20,000 tokens, adjusted to model limits. Repeat queries at fixed concurrency and record median latency, P95 latency and run-to-run variation.[7][9][10][12]

Measure submission-to-completion latency, compression overhead, answer-model latency and completed requests per second. Track answer-output length alongside correctness, retention of qualifications and citation support. Log the hardware, serving region, model version and cache conditions. Shorter input does not guarantee a faster or cheaper pipeline.[7][9][10][12]

Check facts, qualifications and citations

After compression reduces token use, check that the remaining evidence still supports the answers. Evidence preservation is a release criterion. Run the same queries against the original evidence, extractive trimming and generated summaries. Use reference answers and supporting passages to assess the results, and report regressions separately from improvements. If the full-context answer cannot run, label the test as an evidence-preservation check - not a test of answer equivalence.[12][15]

Test retained claims and answer accuracy

Split reference answers into atomic claims: individual statements that can be checked. Compare entities, numbers, dates, units, negation, conditions, exceptions and uncertainty with the compressed evidence. For summaries, also look for changed attribution and false causal links. For trimming, inspect deleted passages for lost qualifications.

Score correctness, completeness and faithfulness separately. A correct headline answer can still leave out an exception that changes the result. Mark each claim as retained accurately, omitted, distorted, contradicted or unsupported.[12][13]

When evidence is incomplete, abstain rather than infer missing conditions.[13][14]

Check that cited passages support each claim

A working link does not prove support. Check that the evidence remains in the retained context and that the cited passage supports the claim’s full scope. Keep document IDs, versions, locations and links separate from summary text. Combining sources must not transfer one source’s claims or authority to another.[16]

Apply the same checks to trimming and summaries, then compare support, scope and traceability.

Review high-risk claims by hand

Pair automated checks with manual sampling of legal, financial, medical, policy and business-process claims. Classify failures as critical, major or minor. Set release thresholds upfront: zero critical unsupported claims and full citation coverage for legal or safety-critical claims. Block critical failures and fix major ones.[12][13][17]

Conclusion: Choose compression that meets quality limits

Optimise pipeline cost, not context length. Compression overhead can wipe out savings at the answer stage. Compare net costs in £ and latency using the same workload, and check that the evidence supports the answers.[18][21][22]

Even exact wording can fail when qualifiers are lost.[3] And a valid citation doesn’t always support the claim it accompanies.[24][25]

Choose trimming, summaries or both

Use the measurements above to choose the least compression that meets your quality limits.

Use extraction when you need exact wording, figures and audit trails. For long passages or evidence spread across sources, use summaries only when every claim maps back to source spans. In hybrid pipelines, trim first, then summarise what remains.[19][20][23] Base your choice on measured quality and net savings.

Turn those measurements into pass/fail limits before deployment.

Set quality thresholds before cutting costs

Keep an uncompressed reference for evaluation. For each workload, set minimum thresholds for claim retention, faithfulness, answer correctness and citation validity. Reject configurations that exceed your limits for factual errors or source-tracing failures, even if they save money. Re-run regression tests whenever models, prompts, chunking or compression settings change.

Test the workload patterns below only after those quality thresholds are met.

Deploy only configurations that preserve claim support and meet your measured cost and latency limits.

FAQs