Name	Date modified
Python.exe: run code right here on the blog	2026-07-28
Math on this blog	2026-07-27
Hello, World	2026-07-26
Embedding drift: Part 3 - Drift is an API break, versioning embedding spaces	2025-11-29
Embedding drift: Part 2 - How are they trained?	2025-11-27
Embedding drift: Part 1 - What is an embedding?	2025-11-26
Linear Vector Spaces and Metrics: What They Are and Why They Matter	2025-11-25

Embedding drift: Part 3 - Drift is an API break, versioning embedding spaces

November 29, 2025 · 11 min read

This is Part 3 of a series on embedding drift. Part 1 covered what embeddings are, Part 2 how they're trained.

The upgrade looks like a one-line diff: embedding_model: v2. No schema change, no code change, tests are green. And yet next morning your app is Part 1's forgetful librarian: shelving baking books under astronomy, retrieving last quarter's memo for this quarter's question, deduplicating documents that were never duplicates.

Nothing in your code changed. But you didn't change a config value; you changed the coordinate system of your product's memory. This post is about treating that with the ceremony it deserves.

TL;DR

A stored embedding is a promise: these coordinates mean something in this model's map. Swapping the model breaks that promise for every vector you've persisted. It's a breaking change to a public interface, and your version number should say so.
The "interface" is the whole embedding pipeline (weights, pooling, instruction prefixes, normalization, chunking), not just the model name. Any change to any of it is a new version.
Measure drift at three levels: geometry (did the map change shape?), behavior (did neighbors change?), outcomes (did answers change?). Only the last one pays the bills, and naive geometric metrics are actually invalid across model swaps, courtesy of Part 2's rotation argument.
Roll out like any API migration: dual-write, shadow-read, canary, rollback. And recalibrate every hardcoded similarity threshold; absolute cosine values do not transfer between models.
The rotation argument also gives you a tool: a learned linear bridge (orthogonal Procrustes) from old space to new can smooth migrations for a surprisingly low price.

The contract you didn't know you published

If your system stores embeddings, your embedding function is a contract:

You persist $z = E_{v} (x)$ , where $E_{v}$ is embedding pipeline version $v$ .
Retrieval, clustering, dedup, routing, and "memory" all consume nearest-neighbor structure in that $z$ -space.

Every vector in your index is a callback into $E_{v}$ . When you upgrade $E_{v} \to E_{v + 1}$ , every one of those stored promises is void: the coordinates still parse (same dtype, maybe even same dimension), but they no longer mean anything in the new map. In software terms: same wire format, different semantics. The most dangerous kind of break, because nothing crashes.

One sharpening that saves real-world pain: the version is the pipeline, not the model. All of these produce a different $E$ , and each deserves a version bump:

model weights (obviously), but also
pooling strategy (mean vs. CLS vs. last-token),
instruction prefixes (query: / passage:; embedding without the prefix the model was trained with is a silent version change),
normalization (did you L2-normalize before storing, or not?),
chunking and truncation rules (different chunks are literally different inputs),
tokenizer or preprocessing changes.

Teams that version "the model" get burned by everything else on that list. Hash the pipeline config; store it next to every vector.

Minimal formalization

Just enough notation to make the metrics precise. The pipeline is a map (in both senses):

E_{v} : X \to ℝ^{d_{v}}, z_{v} (x) = E_{v} (x),

retrieval scores by cosine similarity,

sim (a, b) = \frac{a \cdot b}{‖ a ‖ ‖ b ‖},

and behavior depends on top-k neighbor sets $N_{v}^{k} (x)$ . Drift matters exactly when $N_{v}^{k} (x) \neq N_{v + 1}^{k} (x)$ : different retrieved context means different answers, different memory recall means different downstream decisions. The vectors are an implementation detail; the neighbor structure is the API.

What to measure

Three levels, from cheap-and-weak to expensive-and-decisive. You want all three, but never confuse which level a number lives at.

Level 1: geometry - did the map change shape?

Here's a trap the rotation argument from Part 2 sets for the unwary. The obvious metric is anchor cosine drift: pick a fixed anchor set $A$ and average $\cos (z_{v} (x), z_{v + 1} (x))$ over it. But that compares coordinates across two different maps, and we know two runs can encode the same relative geometry in arbitrarily rotated coordinate systems. For a genuine model swap, anchor cosine can read near zero while retrieval behavior is nearly unchanged, or read high while behavior breaks. Use it only for incremental retrains of the same space lineage (continued fine-tuning of $v$ ), where coordinates are actually comparable.

The geometric metric that is valid across spaces compares relative structure: second-order drift. Sample anchor pairs $(x, y)$ and compare each space's own opinion of them:

{drift}_{2^{n d}} = 1 - corr (\cos (z_{v} (x), z_{v} (y)), \cos (z_{v + 1} (x), z_{v + 1} (y)))

Each cosine is computed within one space, so no cross-map comparison ever happens; you're asking whether the two models agree about which things are similar, which is rotation-proof and exactly the question you care about. (This is the small-scale cousin of representation-similarity measures like CKA.)

Level 2: behavior - did the neighbors change?

For each anchor, compare the retrieved sets directly:

stability@ k (x) = \frac{| N_{v}^{k} (x) \cap N_{v + 1}^{k} (x) |}{k},

averaged over $A$ . This is the workhorse: cheap to compute offline against both indexes, and it measures the thing your application actually consumes. Low stability@k means your RAG context, dedup decisions, and memory recalls are about to change wholesale. If ordering matters to you (it usually does; the top slot feeds the prompt first), also check a rank-weighted variant so that swapping neighbors 1 and 10 doesn't score the same as a stable list.

One more behavioral landmine: absolute thresholds. Every hardcoded number of the form "similarity above 0.9 means duplicate" or "below 0.35 means no match, refuse to answer" was calibrated to $E_{v}$ 's score distribution. Model families differ wildly in how they spread cosine values: some cluster everything in [0.6, 0.9], others use the whole range. Recalibrate every threshold against $E_{v + 1}$ on a labeled sample, or watch perfectly good retrieval get vetoed by a stale constant.

Here's the Level 1 trap and the Level 2 truth in one simulation. Fake a "v+1" by rotating v's space (Part 2's argument says this is realistic) and adding a little genuine disagreement:

import numpy as np
rng = np.random.default_rng(0)
unit_rows = lambda M: M / np.linalg.norm(M, axis=1, keepdims=True)

z_v = unit_rows(rng.normal(size=(2000, 64)))                  # "model v"
Q, _ = np.linalg.qr(rng.normal(size=(64, 64)))
z_v1 = unit_rows(z_v @ Q + 0.02 * rng.normal(size=z_v.shape))  # "v+1": rotated + real drift

anchors = np.arange(500)

# Level 1, the trap: anchor cosine across the two maps
print(f"anchor cosine:      {np.mean(np.sum(z_v[anchors] * z_v1[anchors], axis=1)):.3f}")

# Level 1, done right: second-order drift (each cosine computed within one map)
pairs = rng.choice(2000, size=(1000, 2))
s_v = np.sum(z_v[pairs[:, 0]] * z_v[pairs[:, 1]], axis=1)
s_v1 = np.sum(z_v1[pairs[:, 0]] * z_v1[pairs[:, 1]], axis=1)
print(f"second-order corr:  {np.corrcoef(s_v, s_v1)[0, 1]:.3f}")

# Level 2: stability@10
k, overlaps = 10, []
for i in range(100):
    nv = np.argsort(-(z_v @ z_v[i]))[1:k + 1]
    nv1 = np.argsort(-(z_v1 @ z_v1[i]))[1:k + 1]
    overlaps.append(len(set(nv) & set(nv1)) / k)
print(f"stability@10:       {np.mean(overlaps):.3f}")

anchor cosine:      -0.077
second-order corr:  0.976
stability@10:       0.731

Read that carefully. Anchor cosine screams catastrophe (near zero, as if the models were unrelated). The rotation-proof metrics tell the real story: the two models agree almost entirely about which things are similar, and about 7 of every top-10 neighbors survive the upgrade. Trust the wrong metric and you'd either panic over nothing or, in the mirror-image case, ship a break that anchor cosine happened to miss.

Level 3: outcomes - did the product change?

The one that matters. Maintain a regression suite of end-to-end scenarios:

questions with their expected source documents,
queries with their relevant doc IDs,
user journeys with pass/fail criteria.

Run the whole system under $v$ and $v + 1$ and diff: answer correctness, cited-source overlap, tool/expert call patterns, latency and cost. Geometry and behavior metrics tell you where to look; only task-level deltas tell you whether the upgrade is a regression, a wash, or the improvement the model card promised. Drift isn't inherently bad; the new map is usually better on average. The suite is how you find the segments where "on average" hides "worse for you."

Rolling it out: boring on purpose

The playbook is the same one we already trust for API migrations.

Dual-write. On ingestion, compute and store both $z_{v} (x)$ and $z_{v + 1} (x)$ . Yes, that's double embedding compute and double vector storage. For the migration window, not forever. For the backlog, re-embed in priority order (hot documents first) rather than all-at-once; a lazy "re-embed on next read" strategy can spread the cost further.
Shadow-read. Serve production from $v$ ; in the background, run the same retrievals against $v + 1$ and log the comparisons: stability@k, would-have-retrieved sources, downstream answer deltas on sampled traffic. This is where the Level 2 and Level 3 metrics stop being estimates and become measurements of your workload.
Canary. Route a small slice of traffic to $v + 1$ end-to-end. Watch the regression suite metrics plus the product metrics that no offline suite captures. Roll back on regression, which is trivial because $v$ 's index is still live.
Retire $v$ only when the canary holds and the backlog is re-embedded. Then delete the old vectors; don't leave a mixed-version index lying around for someone to query into nonsense.

This turns "pray and reindex" into a measured rollout with an undo button.

The compatibility bridge

Sometimes you can't dual-write: the corpus is too big, ingestion is out of your control, or you're stuck serving old clients that embed with $E_{v}$ . Part 2's rotation argument suggests a remarkable escape hatch: if a large share of the difference between two spaces is same relative geometry, different orientation, then a linear map should recover most of the compatibility. Learn $W$ on an anchor set:

W = \underset{W}{arg min} \sum_{x \in A} ‖ W z_{v} (x) - z_{v + 1} (x) ‖^{2} .

This is plain least squares, and it handles dimension changes ( $d_{v} \neq d_{v + 1}$ ) for free. If dimensions match and you constrain $W$ to be a rotation (which is exactly the transformation family training leaves undetermined), the problem is orthogonal Procrustes and has a closed-form solution: stack the anchors into matrices $Z_{v}, Z_{v + 1}$ , take the SVD $Z_{v}^{⊤} Z_{v + 1} = U Σ V^{⊤}$ , and set $W = U V^{⊤}$ . No training loop, one linear-algebra call, and the constraint acts as regularization against overfitting the anchor set.

Then old-model queries can search the new index: embed with $E_{v}$ , map through $W$ , retrieve in $v + 1$ -space.

Continuing the simulation from above, build the bridge and measure what it buys:

# without the bridge: old queries against the new index
no_bridge = []
for i in range(100):
    nb = np.argsort(-(z_v1 @ z_v[i]))[1:k + 1]
    nv1 = np.argsort(-(z_v1 @ z_v1[i]))[1:k + 1]
    no_bridge.append(len(set(nb) & set(nv1)) / k)
print(f"stability@10, no bridge:  {np.mean(no_bridge):.3f}")

# the Procrustes bridge: one SVD on the anchor set
U, _, Vt = np.linalg.svd(z_v[anchors].T @ z_v1[anchors])
W = U @ Vt
print(f"cosine after bridge:      {np.mean(np.sum((z_v @ W) * z_v1, axis=1)):.3f}")

bridged = []
for i in range(100):
    nb = np.argsort(-(z_v1 @ (z_v[i] @ W)))[1:k + 1]
    nv1 = np.argsort(-(z_v1 @ z_v1[i]))[1:k + 1]
    bridged.append(len(set(nb) & set(nv1)) / k)
print(f"stability@10 via bridge:  {np.mean(bridged):.3f}")

stability@10, no bridge:  0.003
cosine after bridge:      0.987
stability@10 via bridge:  0.813

Without the bridge, cross-space search returns essentially random results (0.003 is what "mixing maps" looks like in numbers). One SVD later, old vectors land within a whisker of their new-space positions and recover most of the retrieval behavior. The gap that remains, from 0.813 to a perfect 1.0, is the genuine disagreement the noise term injected; that's the part no rotation can undo.

Honesty about the limits: the bridge recovers the part of the change that is orientation, and approximates the rest. Where the models genuinely disagree about relative geometry (the new model learned distinctions the old one never made), no linear map can help, and those are often precisely the neighborhoods you upgraded for. Measure the bridge with the same stability@k and task-level metrics as any other candidate. It's a migration smoother, not a permanent adapter: budget its retirement from day one.

Why this compounds in modular and MoE-style systems

In modular architectures (Mixture-of-Experts routing patterns, agent systems with tool selection, anything with a gating step), retrieval output feeds decisions, not just prompts. Retrieval fills the context, the context influences the router or gate, the router picks a different expert or tool, and the entire computation path changes.

A drift that would cause a mild relevance dip in a flat RAG system becomes a discrete behavioral fork here: one reshuffled neighbor flips a routing decision, and now two "identical" requests take different paths through your system. Embedding drift graduates from a retrieval-quality issue to system-level nondeterminism, unless the embedding version is pinned, measured, and rolled out like the interface it is.

The upgrade playbook

The whole post as a checklist:

Version the pipeline, not the model: weights, pooling, prefixes, normalization, chunking. Store the version with every vector.
Never mix versions in one index or one comparison. (Part 2 proved this is meaningless, not just risky.)
Build an anchor set and a task-level regression suite before you need them.
On upgrade: second-order drift for geometry, stability@k for behavior, task deltas for truth.
Recalibrate every similarity threshold against the new model.
Roll out via dual-write, shadow-read, canary, retire, with rollback live until the end.
Consider a Procrustes bridge for the migration window; measure it like any candidate; plan its retirement.

Closing thoughts

The through-line of this series: an embedding model is a frozen judgment about what counts as "similar" (Part 1); that judgment is manufactured from training pairs and objectives, down to a coordinate system that even the training run itself doesn't control (Part 2); and therefore swapping the model rewrites a public interface of your system. Silently, with the wire format intact.

Nothing here is exotic. Version the interface, measure the break at the level that matters, roll out with an undo button, bridge where you must. It's the same discipline we already apply to every other contract in production, extended to the one contract most systems never wrote down. Do that, and "upgrade the embedding model" stops being a scary ticket and becomes what it should have been all along: a routine migration.

C:\Blog