Changing embedding model or dimensions on an existing data dir

Operator runbook for the one migration CortexDB refuses to do by accident: moving a populated data dir from one embedding model/width to another. Design and evidence: docs/design/REEMBED_MIGRATION.md.

Read the cost first. From the moment the rebuild drops the indexes until a refill completes, the store serves no vector recall over the dropped families — BM25 only, for as long as it takes to re-embed the corpus. Writes keep working; /v1/recall keeps answering, on lexical evidence alone. There is no online path in this release (that is section 3 of the design, unbuilt). Plan the window.

Which situation are you in?

Both startup messages are load-bearing and neither is a nuisance. Their main job is catching a lost API key silently demoting a real corpus to mock vectors. If you did not intend to change the embedding configuration, fix the configuration — do not reach for a flag.

Dimension change: drop and rebuild

CORTEX_EMBEDDING_REBUILD_VECTORS=1 authorizes one destructive rebuild: every vector index whose on-disk width no longer matches the configured model is deleted and recreated empty at the new width. There is no selective re-embed and there never will be — nothing in the store records which provider produced a given vector row, so there is no predicate to select on (design §4.1).

Procedure

1. Back up the data dir. scripts/cold_backup.py with the server stopped. This step destroys vectors; the WAL is the authority and every vector is rederivable from it, but a backup turns a bad afternoon into a restore.

2. Stop the server.

3. Restart once with both flags:

   CORTEX_EMBEDDING_MODEL=text-embedding-3-large \
   CORTEX_EMBEDDING_DIMS=3072 \
   CORTEX_EMBEDDING_ALLOW_REPIN=1 \
   CORTEX_EMBEDDING_REBUILD_VECTORS=1 \
   ./cortexdb
   

   ALLOW_REPIN authorizes the pin change; REBUILD_VECTORS authorizes the deletion. Both are required, and each says something different.

4. Read the banner. The drop announces itself before it happens:

   ================================================================================
     CORTEX_EMBEDDING_REBUILD_VECTORS=1 — DESTRUCTIVE VECTOR REBUILD
   ================================================================================
     DROPPING (every vector in these indexes is deleted): chunk (/data/vector, 1536-dim), …
     Recreating them EMPTY at 3072-dim for provider openai-http:text-embedding-3-large:3072.
     Vector recall will return NOTHING from these families until a refill
     completes — BM25 only, for as long as the corpus takes.
   

   If you do not see it, nothing was dropped — check whether the flag was already spent (below).

5. Refill chunk vectors:

   curl -X POST localhost:3141/v1/admin/index-audit \
     -H 'content-type: application/json' -d '{"repair":true,"vectors":true}'
   

   Synchronous, one embedding batch per event, and idempotent and resumable: detection is "is a row present", so a re-run picks up exactly what is still missing and a crash mid-repair costs progress, not correctness. Re-run until missing_vectors_total is 0. On a large corpus this is a real invoice — it re-embeds every chunk.

6. Re-derive the other families. See the next section. This is the step most likely to be skipped, and the audit will keep telling you about it until you do it or accept it.

7. Confirm. A final {"repair":true,"vectors":true} call retires the rebuild record once every dropped family is populated again, and only then does the audit report "clean": true.

The flag is spent after one use

The first boot that honors the flag writes <data_dir>/embedding_rebuild.consumed. While that file exists, CORTEX_EMBEDDING_REBUILD_VECTORS=1 authorizes nothing — a later width change hits the ordinary refusal. This is deliberate: a flag left behind in a systemd unit or compose file must not lie in wait for the next config typo or upstream model rename. To authorize another rebuild, delete the marker file as a separate, deliberate act.

The flag is also spent when it finds nothing to drop, for the same reason.

What the refill does and does not cover

There are four vector families, each with its own dimension guard, and /v1/admin/index-audit rebuilds exactly one of them.

The audit's repair re-derives an event's chunks from the WAL and re-embeds them. It never writes the other three families, so a clean chunk audit is not evidence that they are back.

The chunk row is partly, not yes, and the distinction has teeth. Enrichment writes fact, triple and summary vectors into the SAME chunk index, and repair re-derives event chunks only. A measured store went 43 rows -> 15 across a drop and a complete repair — one per event — and the audit then reported missing_vectors_total: 0, because nothing in the store records that the other 28 rows ever existed. Re-run enrichment to rebuild them; fact dedup means doing so duplicates nothing. Treat a clean chunk audit as "every event has its chunk vector", which is what it actually proves, and not as "this index is whole". Since v0.9.12 the audit reports all four on every call (vector_families), and a family that is empty while the store still holds the rows that should have populated it is reported as empty_but_expected — the audit will not answer "clean": true over it.

The derived family cannot self-heal after a width change. The bounded online backfill replays the persisted fp16 layer-vector rows into the derived index; after a width change those rows are the OLD width and every one of them is skipped (they are not inserted at the wrong width — that would seed the new index with unusable rows). Fact and belief embeddings come back only by re-running the fact/layer builds, which re-embed from the fact text.

If your deployment does not use entity resolution, derived layers or rollup summaries — enrichment is off by default, and so are derived embeddings and rollup summaries — those families are legitimately empty and the audit says so (expected_at_least: 0, empty_but_expected: false). The rebuild record still names any family whose directory was dropped; delete <data_dir>/vector_rebuild_pending.json to accept the store as it is.

Same-width switch: the silent one

Changing provider or model without changing the width trips no index guard at all. The vector index opens, BM25 answers, recall returns results — and every one of them is scored in the old embedding space against queries embedded in the new one. CORTEX_EMBEDDING_ALLOW_REPIN=1 is enough to boot, and nothing re-indexes.

There is no automatic detection: no vector row records which provider produced it, and the repin has already overwritten the only provider identity in the store. What there is instead:

• the repin writes <data_dir>/embedding_provider.repin.json, and
• /v1/admin/index-audit reports it as embedding_repin on every call, with the remediation spelled out, until you delete the file.

Because the width did not change, CORTEX_EMBEDDING_REBUILD_VECTORS finds nothing to drop. To re-embed deliberately, stop the server, delete the vector index directories listed in the table above by hand, restart, and follow steps 5-7. Then delete the repin marker.

Recovering from a refused boot

A boot that is refused changes nothing. Since v0.9.12 the pin is written only after storage initialisation succeeds, so a dimension guard that refuses the boot leaves the data dir exactly as it was and the store still starts on its previous configuration with no second ALLOW_REPIN. (Before that, a failed dimension migration rewrote the pin and then refused, stranding the data dir on a configuration that could not serve it.)

If the rebuild itself fails partway — a directory that cannot be deleted — nothing is recorded as consumed and the boot is refused, so fixing the permissions and restarting retries the whole thing.

Files this leaves in the data dir