How to Detect RAG Index Drift After Document Changes

I detect RAG index drift by comparing a source snapshot with the published index - not by checking timestamps alone. I match document IDs, revisions and hashes, then check that every expected chunk is present and that deleted content is no longer searchable.

My checks cover:

• Documents and chunks: Find missed edits, missing or duplicate chunks, mixed revisions and stale permissions.
• Embeddings: Check the model, settings and exact input hash. Current metadata does not prove a vector is current.
• Retrieval: Test whether changed content appears in the top five results and superseded revisions stay out.
• Repair and monitoring: Fix the failed stage, validate replacements before switching traffic, and track drift rates and update delays.

I treat changes within the normal ingestion window as in flight, not drift. The release rule is simple: check both stored records and what users can retrieve.

::: @figure {How to Detect and Repair RAG Index Drift} :::

Embedding Drift: The Silent Bug Killing Your RAG Pipeline

Build Source and Index Manifests

Use the source manifest as the authoritative record of what should be searchable. Create one row per document and one row per indexed chunk, including deletions. Build the source manifest from a database snapshot, repository commit, change-data-capture position or export identifier. Record that boundary and its UTC cut-off, then export the index manifest from the corresponding published generation.[5][6][7] These manifests give you a baseline for finding missed edits, partial updates, deleted content and stale embeddings.

Changes within the documented ingestion window are in flight, not drift. Retain snapshot_id, observed_at, revision, ingestion_run_id and index_generation, and exclude edits made after the cut-off. If a true snapshot isn't available, keep the revision record and ingestion start and end times. Then rerun the comparison for records changed during ingestion.[7][8]

Match Document IDs, Revisions and Index Fields

Join records using a canonical document_id, and store the original source key separately. IDs establish identity, revisions establish version, and hashes confirm content equality.[3][4][5]

¹ Generate these in a derived expected-chunk manifest using the applicable pipeline configuration.[3][6][9]

With the records aligned, you can run the remaining checks as simple mismatch tests.

Apply Consistent Document and Chunk Hashing

After checking identity, verify that both sides normalised and chunked the content in the same way.

Define a versioned canonical representation before hashing. Convert CRLF and CR line endings to LF, remove an optional byte-order mark and sort unstable metadata keys. Document any rules for trailing whitespace or Unicode. Keep meaningful headings, table values, code indentation and citation page numbers: removing them can hide actual changes.[8][10]

Keep the document hash, then reproduce the expected chunks from that source revision. Record the extractor, OCR settings where applicable, normalisation version, chunking algorithm, size, overlap and separator rules. Derive chunk IDs deterministically from the document ID, revision, ordinal and chunking configuration.

Compare document hashes only with document hashes, and expected chunk hashes only with indexed chunk hashes - never document hashes with chunk hashes. Use the same extractor and chunking settings on both sides. Configuration drift can change chunk boundaries even when the source hasn't changed.[3][5][6][8][9]

Compare Records to Find Failed Updates

Once the source and index manifests are aligned, compare records by document_id, access_scope and revision state to find failed updates. Use a full outer join on document_id and access_scope so rows present on just one side remain visible. Flag source-only rows as missed ingestion. Before treating index-only rows as drift, check deletion tombstones, historical generations and scope mismatches.[5][6][4]

Interpret Revision and Hash Mismatches

Revision and hash mismatches are symptoms, not proof. A revision change may affect metadata only. A hash mismatch matters only if both sides use the same canonical input and algorithm.[5][6][11]

Use these patterns to separate actual drift from harmless metadata changes.

Detect Missing, Mixed and Outdated Chunks

Once document-level checks pass, check chunk membership and whether each chunk is current.

Compare chunk membership, not counts. Group active chunks by access_scope and document_id. The difference expected − active identifies missing chunks; active − expected identifies unexpected chunks, including orphaned chunks from deleted passages. Check duplicates separately. Then compare each chunk’s revision, document hash, chunk hash, ingestion run and generation against its expected entry to find mixed or outdated content.[1][4][7]

Use the ingestion registry to tell deliberate reuse apart from incomplete processing. A reused vector may keep an earlier embedding timestamp if its input hash and embedding configuration match. Its published record must still map to the intended revision and generation. A document-level updated flag proves neither completeness nor removal of old chunks.[3][13]

Trace discrepancies through extraction → chunking → embedding → publication using the ingestion correlation ID. Check extraction status, text hash, expected chunk entries, embedding input hashes and job status. Require atomic replacement or a generation switch, with no unexpected chunks and with hashes, scope and embedding settings validated. Switch generations only after every expected chunk is present, current and unique.[5][14]

Verify Embeddings and Retrieval

After comparing records, check that stored vectors and retrieval results still reflect the current revision. Check content, embedding and retrieval drift separately. Current metadata does not prove that the vector or ranking is up to date.[16][17][20]

Check Embedding Settings and Input Hashes

Compare each vector’s model, version, dimensions, preprocessing, normalisation and task prefix with the active index contract. Check the query/document model pair and similarity metric as well. Matching dimensions alone do not prove compatibility. Group records by embedding configuration to spot mixed generations, then compare those groups with the intended contract.[15][17][19][20]

Recalculate embedding_input_hash using the exact input sent to the model. Include titles, headings and added context - not just the chunk body. Link that hash to the model request, response status and vector-write acknowledgement. If provenance is missing or contradictory, mark the vector as unverifiable and re-embed it. A model change needs a separate index generation.[15][19][20]

If the vector contract passes, test whether retrieval still returns the current revisions correctly.

Test Retrieval of Current Document Revisions

Keep a fixed set of versioned query–document pairs covering common searches, edited passages and superseded content. Record the expected document IDs, revisions, acceptable chunks and excluded revisions. Set the cut-off before testing, such as Recall@5. Save returned IDs, revisions, chunk hashes and ranks - not just the answer text.[16][17][20][21]

In a non-production index, change a source revision, run the normal pipeline and query the changed topic. Use document IDs, revisions and chunk hashes to check that the current revision appears in the top five results and the superseded revision is excluded. Compare results before and after repair using identical queries, filters, embeddings and reranker settings. If the current chunk still ranks too low, check truncation, filters and approximate-search settings before re-embedding.[16][18][20][21]

Test retrieval separately from answer quality. Fluent answers can hide stale evidence.[16][18][20][21]

Repair Drift and Monitor Index Updates

Reprocess Affected Content and Publish Safely

Once you identify drift, fix the failed stage and republish only the affected revision. If the chunking, preprocessing and embedding configuration still work, leave the rest of the index alone. Match the repair to the failure:

• Missed edits: Fix ingestion or change detection, then replay the revision.
• Partial updates, chunking or metadata failures: Re-chunk and replace the affected chunks.
• Mismatched embeddings or embedding failures: Re-embed the affected content.
• Publication failures: Retry writes idempotently.[23][12]

Publish the replacement revision to a staging namespace or inactive generation using stable IDs such as document_id:revision:chunk_number.

Before publishing, reconcile document and chunk counts, hashes, revisions and embedding provenance. Switch retrieval only after validation, then retire every superseded chunk. Confirm a deletion only when the source record is gone and tombstoned - a temporary fetch failure does not prove deletion. Remove all chunks for confirmed deletions and check that none remain searchable.[12][24][25][29]

If the embedding model, vector dimension or query configuration changes, treat the new setup as a separate, compatible index generation. Validate it first, then switch query embeddings and retrieval traffic together. Keep a rollback target available. For an audit trail, record the repair run, source revision, configuration snapshot, validation results and publication time.[26][28]

Track Drift Rates and Detection Delays

After republishing, check whether drift is falling and how quickly the index catches up. Define the source snapshot and denominator before reporting rates. Track each measure by drift class rather than combining them into one score:

• Source hash match rate: Use source documents as the denominator.
• Indexed-chunk completeness (missing-chunk rate): Use expected chunks as the denominator.
• Unexpected/duplicate-chunk rate: Use indexed chunks as the denominator.
• Embedding provenance rate: Use vectors as the denominator, checking the required model version, dimension and input hash.[22][26]

Report counts alongside percentages, and record inaccessible sources separately. These measures should confirm that the source, chunks and embeddings now match.[22][26]

Monitor the age of the oldest stale or unverified record, source-change-to-index delay, detection delay and current-revision retrieval hit rate. Set alerts against the update deadline and normal ingestion window - not every short queue delay. Break results down by connector and stage. Alert separately on failed deletions and revisions still absent after publication, so you can distinguish slow processing from updates that never completed.[23][25][27]

Conclusion: Use the Index Validation Checklist

After repair, run this checklist before publishing the new generation. Treat it as your release gate: build source and index manifests, join them by stable document_id, then compare revisions and normalised content hashes. Check the full expected chunk set for missing, duplicate, mixed-revision and superseded chunks.[30][33]

Validate each vector’s embedding model, version, dimensions, preprocessing and input hash. Run versioned retrieval tests for added, changed, removed and unchanged facts. Record the returned document IDs, revisions, chunk IDs and evidence.[31][32]

Recent timestamps and document-level ‘processed’ flags do not prove the index is up to date. Metadata shows what was indexed; retrieval shows what users can retrieve. You need both checks.[29][30]

FAQs