RAG Index Versions: How Embedding Changes Affect Rollback

I treat RAG rollback as restoring a retrieval stack - not just an index. Before switching traffic, I check that the saved index, query encoder, search settings and filters work together. Matching vector dimensions alone does not prove compatibility.

My rollback plan covers four things:

• Version the bundle: Pin the corpus, encoders, processing rules, search configuration, reranker and dependencies.
• Test before switching: Build separately, reconcile document changes and compare retrieval quality, latency and access controls. For example, a promotion gate could limit recall@5 loss to 1 percentage point.
• Switch and verify: Route each request to one tested bundle, isolate caches and check results - not just HTTP status codes.
• Keep recovery safe: Retain working releases, name owners and rehearse restoration. Rolling back an index must never restore deleted records or revoked permissions.

::: @figure {Safe RAG Index Rollback: Version, Test, Switch, Recover} :::

Your RAG Pipeline Needs Reindexing. How Do You Do It Without Downtime? | GenAI Interview Prep

What to Include in an Index Version

Rollback works only when the index, encoders and search settings are versioned together. Treat the manifest as the rollback contract for the retrieval stack.

Give every release a permanent ID, build ID, creation timestamp and release status. Once promoted, never edit it in place. Publish index-v48 instead. Pin dependencies to exact revisions or file hashes in the manifest, not moving labels such as latest.[2][1][4]

Record Data, Encoders and Search Settings

Use this manifest structure for every indexed release. Keep large inventories, checksums and evaluation reports in immutable files, and reference them from the manifest.

Validate the manifest before activation. Reject missing artefacts, dimension mismatches, unsupported metrics, unapproved encoder pairs and incompatible filter schemas. Check model hashes and index completeness, then run known probe queries.

Fail closed at startup rather than serve an unchecked combination. If validation fails, stop the rollback before any traffic reaches that release.

Decide Which Changes Require a Rebuild

Not every release needs new document vectors. Group changes by what they affect: embeddings, the search index or serving behaviour.[3][5][6]

Use this matrix to decide whether to rebuild, redeploy or just change the configuration.

For query-only changes, test that the new encoder still works with the stored vectors. Matching dimensions alone isn't enough. Use labelled retrieval tests and score-distribution checks. Version the search configuration too: rollback must restore behaviour, not just files.[3][5][6]

Why Embedding Changes Can Break Rollback

Restoring an older index pointer does not restore compatibility if the active encoder, search settings or cache still belong to a newer release. The main rollback risk is compatibility drift, not file restoration alone.[3][1][10]

Spot Errors and Retrieval Quality Loss

Failures are usually silent. Embeddings with the same dimensions can occupy incompatible spaces, so searches still run while relevance and ranking drift. Changes to metrics or normalisation can invalidate score thresholds. A lower ANN search budget can also reduce recall, even when the encoder works correctly. Cosine and dot-product only match on unit-length vectors.[7][8][9]

Schema changes can break filters, pagination and access control. Caches keyed only by query text can return results from the wrong release. Keep incompatible vectors in separate indexes or namespaces, with explicit routing.

Check Document and Query Encoder Pairing

Document and query encoders can be separate components. Their compatibility depends on their trained relationship - not model names or vector length. Validate retrieval quality for the pair as a unit, rather than checking each model independently, before treating it as supported.

Diagnose Retrieval Compatibility Problems

Use a fixed checklist to separate version drift from retrieval faults.

• Verify the active release and inputs: Compare the index, exact encoder revisions, dimensions and data types against the intended release. Check the approved encoder pair, instructions, truncation and preprocessing. Recalculate input hashes from the exact text embedded, including headings.
• Check search settings and retrieval quality: Check normalisation, metric, score thresholds and ANN settings against the release manifest. With queries, documents and top-k held fixed, compare recall@k, nDCG, score distributions, empty-result rates and p95 latency. Compare exact-nearest-neighbour results on a small, fixed subset to isolate recall loss caused by approximation.
• Inspect cache namespaces: Separate query embeddings and search results by retrieval version, and invalidate incompatible entries.
• Test filters: Check that permissions, field types, cursors and pagination remain compatible with the restored release.
• Check reranking: Verify the release’s reranker revision, input format and candidate count.
• Separate generation changes: Compare retrieved chunk IDs and their order before inspecting answers. If retrieval is unchanged, investigate context assembly, truncation, prompts and the generation model.

Build, Test and Promote a New Index

With the manifest fixed, build green as a separate release candidate while blue continues serving production. Build and test green separately. Keep each release’s configuration immutable, and track document updates explicitly.[11][12][13]

Build Separately and Track Document Changes

Record the corpus snapshot, source revision, extraction time, document and chunk counts, checksums, deletion markers, ACLs and exact embedding versions.

Build green from source documents rather than reusing vectors. Embed each release with its own pinned document encoder so it stays compatible with its retrieval stack. Backfill separately, then replay creates, updates and deletions made after the cut-off before testing.

Dual writes can cut catch-up work, but keep a durable log for failed writes. Reconcile document IDs and revisions, chunk counts, metadata, access-control filters, tenant boundaries and deletion markers. Matching total vector counts is not enough.[18][20]

Set Measurable Promotion Criteria

Once green is rebuilt and reconciled, test it against blue using the same labelled set. Send shadow production queries to green without returning its responses to users. Group evaluations by business query type, and hold the generation model, prompt, temperature and context limit fixed. Tag results with the versions of the index, document encoder, query encoder, retrieval configuration and evaluation set so regressions can be reproduced.[14][16][17]

Set the promotion gates in advance. For priority query groups, recall@5 must be no more than 1 percentage point below blue, and nDCG@10 must at least match blue. Groundedness, citation correctness and completeness must meet agreed targets. Keep p95 latency, resource use and cost per successful answer within budget, and check errors, timeouts and empty results separately.

A freshness target could require 95% of changes searchable within five minutes, with explicit deletion and permission-change deadlines.

Any access-control or tenant-isolation failure is a hard stop, regardless of average relevance gains.[14][15]

Choose a Rollout and Switch Routing Safely

After green passes the gates, choose the cutover pattern.

Switch a versioned routing bundle, not just an alias. Bind each request to one physical index, query encoder and retrieval configuration, and record the document-encoder version for traceability.

Make the routing update atomic. Record the approver, time and traffic weights, and retain blue throughout the agreed monitoring window. An alias that changes independently of the encoder cannot provide a safe cutover.[12][13][18][19]

Roll Back and Retain Index Versions

Run and Verify the Rollback

If a promoted release performs worse, roll back the whole versioned bundle - not just the index pointer. Freeze index and model promotions, pause non-critical ingestion changes, and appoint an authorised incident lead. The incident lead selects the latest accepted release, the service owner approves the rollback, and security signs off for sensitive records.

Record the incident ID, target release, reason, approver, operator, timestamp, expected risks and success criteria. Before switching traffic, check the index snapshot, manifest, encoder revisions, dimensions, metric, search settings, reranker, build and runtime image. A model identifier alone is not enough: pin the exact callable encoder revision.[21][24][25] Then restore the target release’s exact runtime dependencies and routing state.

Restore the matching dimensions, metric, normalisation, search settings, reranker and filters. Switch traffic through an atomic alias, feature flag or service-discovery pointer only after the target is healthy. Keep the previous routing state available until post-switch checks are complete.

Invalidate query-vector, retrieval-result and reranker caches. Check representative exact-match and semantic queries, latency, empty results, metadata filters, tenant boundaries, denied access and ranking against the stored baseline. HTTP 200 responses alone do not confirm that retrieval works correctly.

Before cleanup, preserve the failed and restored release manifests, index and model hashes, configuration diffs, routing events, deployment logs, model-load errors, access decisions, cache actions and ingestion offsets in access-controlled, tamper-evident storage. Resume ingestion only after defining how to reconcile post-release changes.[22][23]

Retain Versions and Reconcile Document Changes

Document an observation window that covers normal traffic, delayed reports and recovery needs. Treat 7–30 days as a planning range, not a fixed rule; high-impact systems may need longer. Keep the previous index readable, together with its working dependencies and any required application version. Set separate expiry rules for queryable copies, cold snapshots and diagnostic evidence, subject to privacy and retention requirements.

Make ownership clear:

• Platform or search team: index snapshots, aliases, compatibility checks and restoration automation.
• ML team: encoder artefacts and evaluation baselines.
• Application team: client and schema compatibility.
• Security: approval of access policies, secrets, tenant isolation and evidence handling.
• Data governance or legal teams: deletion, retention and residency requirements.
• Service owner: rollback approval.
• Incident lead: execution and communications.[23][24]

Restoring retrieval behaviour does not mean restoring every historical document change. Index rollback is not data rollback. Compare the retained manifest and ingestion watermark with the current source of truth using stable document IDs. Replay only authorised, timestamped changes, such as new or updated documents and permission changes. Use the encoder and schema compatible with the rollback release.

Do not reintroduce documents deleted for legal, privacy or business reasons. Likewise, do not reverse a revoked permission just because an older snapshot still contains the record. If replay requires a rebuild, let the old index serve traffic only where its data-handling and access-control guarantees remain valid.

Rehearse Rollback Before an Incident

Rehearse the rollback path under realistic load before an incident occurs. Test an unavailable hosted-model revision, an outdated index, incompatible vector dimensions, stale caches and an interrupted routing switch.

Measure decision time, restoration time, time to acceptable retrieval quality, document-update gaps and time to resume ingestion. Check automation, monitoring, cache isolation, audit capture and current access restrictions - not just whether the snapshot loads.

Conclusion: Rollback Readiness Checklist

Version the whole retrieval stack, not just the index. Matching dimensions do not prove compatibility. Restore the index, query encoder and search settings together.[26][27]

Use this checklist as the final release gate:

• Complete manifest: Record the corpus snapshot, chunking and preprocessing, encoder revisions, dimensions, metric, filter schema and search settings.
• Reproducible build: Use immutable snapshots, pinned dependencies and deterministic preprocessing. Record verification results.
• Promotion criteria: Agree limits for retrieval quality, latency, errors, data freshness and access-control correctness.
• Coordinated cutover: Switch the index and query encoder together in one atomic step.
• Versioned monitoring: Make sure logs, traces, evaluations and alerts identify the release serving requests.
• Retained dependencies: Keep model revisions, tokenisers, packages, service versions and credentials.
• Retained artefacts: Keep encrypted, access-controlled snapshots and rebuild instructions.
• Data safeguards: Apply cascading deletions, permission metadata, retrieval-time authorisation and tenant boundaries.
• Rehearsed rollback: Name owners, define clear triggers, rehearse document-change replay and measure recovery time.

FAQs