CortexDB 0.10.5 — fast erasure on long-lived stores, no spurious wait=indexed timeouts, cross-customer answers that mean the same in every language, and a Settings page that keeps your keys

0.10.5 is a fix release on top of 0.10.4, the last published release. Everything in RELEASE_NOTES_0.10.4.md still applies; this page lists only what changed since. It adds no column family, directory, WAL row kind, index field, key prefix or on-disk format, and nothing is migrated: a 0.10.4 server opens and serves a data dir 0.10.5 has used (see "Rollback").

Summary

• Erasure and its proof are fast again on long-lived stores. On a store whose old retry records had expired, POST /v1/erasures and GET /v1/erasures/{id}/verify each took 25-60 s for one customer, close to a 100 s proxy origin limit for a larger one. Measured on a copy of a store of about 130,000 events, one customer of 219 events: 24.6 s to erase and 24.8 s to prove on 0.10.4, 0.26 s and 0.02 s on 0.10.5. What an erasure deletes and proves is unchanged.
• ?wait=indexed no longer answers a spurious 408 WAIT_TIMEOUT. A write whose event was already indexed could wait out the whole 30 s budget and answer 408 with waiting_on: "stage". The wait no longer misses the moment the stage completes. The same fix covers wait=consolidated, bulk writes and the lifecycle waits.
• Cross-customer questions in Hindi or Hinglish no longer answer 422 because the model planner wrote a time phrase the server cannot resolve ("all time"). A phrase that means all time is dropped with a note; any other unreadable phrase falls back to the rules planner, so a question about last month is never silently answered over all data.
• "Read every conversation" answers agree across languages and runs, and keep the question's meaning. A plain topical question ("which customers discussed a loan or EMI on a call") reads the same way in English and in Hinglish, and a complete read is reused (in memory only) by a later answer over the same data. A negation, a conjunction, a specific verb or a "why" question reaches the readers as asked (for Hindi and Hinglish, as the planner's faithful English translation), never reduced to its topic words. Details under "Read-all answers".
• Settings: saving a card with its key field empty keeps the stored key, and the "take effect after a restart" banner clears after the restart.
• Label limits are announced, not enforced. 0.10.5 adds limits on context.labels (count, size, empty labels, control characters) that a later release will enforce by default. They are off unless an operator sets CORTEX_LABEL_LIMITS=on. Check your integrations now; details under "Label limits (announced)".
• Docs: the scope types each deployment preset allows (the hosted service allows 10), a warning that a forget without cascade keeps the memories and how to delete them, and directives.extract documented as a list.
• No client release. Every client published with 0.10.4 works against 0.10.5 unchanged.

Upgrade from 0.10.4

Before you upgrade: what clients and operators will notice

Only deployments with the experimental cohort lanes on (CORTEX_COHORT=on, CORTEX_SUBTREE_RECALL=request) see the first four items:

• cohort.read (an answer that read every conversation) has new fields: reused, computed_at, plan_fingerprint, cohort_fingerprint, topic (the plan's canonical topic) and reader_question (the question the readers got, when it is not the asker's own). A reused answer says so (reused: true, the earlier read's computed_at, a note starting result reused:) and reports no map, theme or check call of its own. The plan echo (cohort.plan.query) carries the canonical topic.
• A model-planned answer can carry new notes: planner window '<phrase>' ignored: the question is answered over all data; LLM planner unusable (its window '<phrase>' is not a time phrase the server can resolve); a glossary note ending (operator glossary, binding: added to the LLM plan, which had left it out); and the readers were given the question in English as "..." (the planner's translation).
• A Hindi or Hinglish question on the ranked lane (cohort.read: "sample", or read-all off) also searches with the asker's own words, beside the plan's English query, so its excerpts can differ from 0.10.4's.
• With CORTEX_COHORT_PLANNER=auto or llm, the planner call asks for one more field (reader_question, about 20 output tokens).
• The internal Settings route behaves differently (PUT /internal/ui/settings/{id}, used by the bundled console): an API key sent blank or null is kept and listed in a new kept field of the response; only "clear": ["api_key"] removes one, and sending a value and a clear for the same field answers 400 BAD_REQUEST. Connections report pending_fields. The console in the same binary is updated to match. A script that sent "" to remove a key now keeps it silently (the response lists it under kept): send clear instead.
• If a 0.10.3 console ever saved a card with its key field empty, that key is still missing after the upgrade. That console wrote an override that removes the key at every boot, and 0.10.5 does not rewrite your settings_overrides.json: it logs SETTINGS_OVERRIDE_CLEARS_SECRET naming the entry. Re-enter the key in Settings (or delete the entry from the file) and restart.
• One new boot log line says whether the label limits are enforced (label limits: not enforced by default).

Nothing else changes for a client: no new route, no new required field, no changed default response.

Upgrade steps

Nothing is migrated and there is nothing to run afterwards. Replace the binary or image and restart as for any release, giving the server 120 s to stop. Then check:

1. GET /v1/admin/version: crate_version 0.10.5 and the git_sha you deployed. The capability list is the same 11 names as 0.10.4.
2. GET /v1/admin/ready: ready: true.
3. Search the boot log for SETTINGS_OVERRIDE_CLEARS_SECRET (see above).
4. Before a later release enforces the label limits, check your integrations' labels (see "Label limits (announced)"); CORTEX_LABEL_LIMITS=on on a staging server shows what would be refused.

Upgrading from 0.10.3 or earlier

0.10.5 contains everything in 0.10.4. Read RELEASE_NOTES_0.10.4.md, "Upgrade from 0.10.3" (and from there the earlier notes) first; this page applies on top.

Fixes

Erasure and its proof on long-lived stores

An erasure and its proof find the idempotency (retry) records of every event they touch through the idempotency store's event-to-record reverse rows. Those reads were prefix scans without an upper bound. When a store's old retry records expire, the boot clean-up deletes them, leaving one deleted row per expired record, and a lookup whose prefix held no live row walked every one of them before it stopped. On a store with about 130,000 expired records every erase and every proof took 25-60 s. The reads now stop at the end of their prefix, so the same steps take well under a second; a test checks that the bounded reads return exactly what a full scan returns for every prefix shape.

?wait=indexed and the lost stage wake-up

The stage wait read the event's stage first and subscribed to stage notifications only then, and its 500 ms poll re-checked only for failures. When the stages finished between that read and the subscription (on an idle server they finish within microseconds of the write), the notification was missed and the request waited out the whole budget, answering 408 WAIT_TIMEOUT with waiting_on: "stage" and waited_ms near 30,000 for an event that was already indexed. The wait now subscribes first, then reads, and reads again on every wake-up and every poll. A failed stage still fails at once, a real timeout still answers 408 at the deadline, and a 408 with waiting_on: "indexer" (a real indexer backlog) is unchanged. Two deterministic tests reproduce the race: both time out on 0.10.4's code and pass on 0.10.5's. The race is rare under load: about 51,600 wait=indexed writes against the release candidate (sequential and concurrent writers, and a write, seed and forget load) gave no 408, and the same loops gave none on 0.10.4 either, so the tests are the proof, not the load runs. The race matches an intermittent 408 in this project's own CI, which has not recurred in the release runs.

Cross-customer questions in Hindi or Hinglish: time windows

With the model planner on (CORTEX_COHORT_PLANNER=auto or llm), a question such as "kin customers ne kabhi bhi ..." could get a plan whose window.natural was a phrase the executor cannot resolve, and the answer was 422 INVALID_BODY naming plan.window.natural. Now, on a model-written plan only:

• a phrase that means no window at all ("all time", "ever", "overall", "kabhi bhi", "ab tak") is dropped with a note, and the question is answered over all data;
• any other phrase the server cannot resolve (for example "pichle mahine" left in Hinglish instead of "last month") makes the plan unusable, so the rules planner's outcome stands, with a note: a question about last month is never answered over all data.

A plan you send yourself in cohort.plan is checked exactly as before.

Read-all answers: the same in every language, with the question's meaning kept

Asked "which customers discussed X" or "what do customers raise about X" in English and then in Hinglish, the read-all lane could list different customers, and two runs of the same question could too. 0.10.5 changes what the readers (the map, theme and check calls) are given:

• A plain topical question gets one canonical English question. A question that only asks who discussed, mentioned, talked or asked about a topic (themes: what customers raise or say about it), with no negation, no and/both, no other verb or qualifier, and several topics joined only by or, is given to the readers as a question built from the plan's topic: "Which customers discussed loan or EMI?", whatever language it was asked in. Its topic is the plan's query words without function words, question words, generic lookup verbs ("discussed", "ki baat ki") and the words of a channel the plan's own conditions already select ("phone call"), with acronyms kept.
• Every other question keeps its meaning. A negation ("which customers did not mention a refund?"), a conjunction ("discussed both insurance and a loan"), a specific verb ("asked to cancel their loan", "wanted a callback") or a reasons question ("why do customers cancel?") reaches the readers as asked: in the asker's words for an English question, and as the model planner's faithful English translation for a Hindi or Hinglish one (a translation that does not read as English, or lost the question's negation or and, is not used; the asker's words are). These questions never share a cached read with the plain question about the same topic.
• The operator glossary is binding on a model-written read plan. A glossary phrase of the question whose facet the plan leaves out (for example "phone call" left out of a Hinglish question's plan, which would read every chat too) is added to the plan with a note, so the cohort is the one the rules planner builds for the same words in English. A facet the plan already names, also in other terms for the same channel, or whose words the model kept as the topic ("call back"), is left as planned. The planner is also told that a denial of what was said reads the same conversations as the question without it: "Kin customers ne phone call par loan ya EMI ki baat nahi ki?" was planned as "customers who never had a call" (3 conversations read) on an earlier 0.10.5 candidate, and now reads the same 574 calls as the English question.
• A complete read is reused. The result of a read in which every conversation was read and none was erased while it was read is kept in the server's memory and served again, with no map, theme or check call, to a later answer with the same identity: the same tenant, scope, subject level, naming and alias key, the same plan over the same conversations at the same data revision, the same reader question, models, prompts and read settings, and the same caller authorization. Before a result is reused, every conversation it was computed from is checked again against the live forget and erasure fences; one that fails drops the scope's results and the cohort is read again. /v1/forget and /v1/erasures clear the whole cache before they run and when they finish. At most 16 results (32 MiB) are kept, least recently used out; nothing is written to disk and a restart empties it. naming: "alias_ephemeral" answers are never kept. The narration still runs for every answer, in the asker's language. CORTEX_COHORT_READ_CACHE=0 turns reuse off.

Measured on a copy of a 300-customer store whose cohort of phone calls holds 574 conversations (read-all on, map model claude-haiku-4-5, answer model claude-opus-4-8). On 0.10.4 the question "Which customers discussed a loan or EMI on a phone call?" listed 115 to 128 customers from one run to the next, and its Hinglish wording 132, because the model planner left out "phone call" and the chats were read too. On 0.10.5:

A fresh read of the 574 calls took 12-17 s and cost $1.24-1.28 in model calls; a reused one 6-10 s and $0.04-0.07 (the planner and the narration). The two fresh reads of the plain question (113 and 112 customers) show the run-to-run spread that reuse removes for repeated questions.

Reuse needs the same data revision, so on a scope that receives captures all the time a result is rarely reused; the canonical question and the glossary binding are what make two such reads agree. docs/ERASURE_COVERAGE.md lists the result cache as a store (no conversation text; covered by the clears and the fence re-check).

Settings page

• A blank key field keeps the stored key. The 0.10.4 console already left a blank key out of a save, but the server still read a key sent as "" or null as "remove it", so a client that sent the field (the 0.10.3 console did) lost the key at the next restart. A blank or null key is now never written; the card's "remove it on save" box (clear) is the only way to remove one. A key an older console already removed stays removed until you re-enter it (see "Before you upgrade").
• The restart banner clears after the restart. A connection is pending only while a saved value differs from the value the running process booted with, not whenever an override exists; pending_fields names the fields. The console no longer keeps its own "saved this session" flag.

Label limits (announced)

Before 0.10.5 nothing checked a label. 0.10.5 adds these limits on context.labels, not enforced by default: a cap on what clients send is announced before it applies. A later release will enforce them by default; an operator can enforce them now with CORTEX_LABEL_LIMITS=on.

• Turning them on: CORTEX_LABEL_LIMITS=on (also 1, true, yes, enforce) in the server's environment, then restart. The boot log says label limits: enforced on new writes.
• Raising or lowering them: CORTEX_LABELS_MAX_COUNT and CORTEX_LABEL_MAX_BYTES (integers, at least 1). A value that is not a number, or is 0, keeps the default and is reported at boot (code=LABEL_LIMIT_INVALID). Raising them never breaks a client.
• What is checked while they are on: every new write (POST /v1/experience, every /v1/experience/bulk item, every /v1/import row), after the pre-0.8 metadata.labels alias is folded into context.labels; stored events are never re-checked. details names the field, the label's index and the limit, never the label's text. A bulk batch with one failing item answers INVALID_ENVELOPE with the label code on that item and writes nothing; an import row fails alone.
• Why these numbers: one pilot's ingester writes at most 15 labels of at most 58 bytes. The first-party connectors write one label per entity (a calendar invite carries one per guest) and carry metadata values in key=value labels (a Notion page's raw properties, an email's recipient lists) whose real sizes have not been measured, so the byte default is set far above anything seen. Before enforcement becomes the default it will be checked against real connector stores.
• What to check now: labels that are empty or only whitespace, or that hold a control character other than tab, line feed and carriage return. cortexdb-connectors 0.2.22 writes an empty tag label when a provider leaves a field out (an Intercom conversation's state, part type or role, a Confluence space key, a Zendesk status); a later connectors release drops empty tags. The served schema (/v1/schemas/experience_envelope) states the limits in x-cortex-label-limits, not as constraints, so a client that validates against it accepts what a default server accepts.

API reference §3.5.3, "Label limits", has the full rules.

Docs

• API reference §3.4.1 lists the scope types each preset allows: cloud_shared_saas (the hosted service) allows org, dept, team, app, user, agent, service, ws, project and source; on_prem_enterprise and cloud_private add global and system; dev_local adds debug and temp. §5.3.2 now shows the cloud_shared_saas preset as the server loads it.
• API reference §8.5 warns that a forget without cascade runs derived_only on every deployment, the hosted one included, and keeps the memories; it shows how to delete them (selector.memory_ids, /v1/erasures) and which deletions free the writes' idempotency keys (a memory_ids forget and an erasure do; a scope-wide redact_events forget does not).
• directives.extract is documented as a list of stage names: omitted or null runs every stage, [] runs none, and false or a string is 422 INVALID_BODY. It is honoured on single and bulk writes (since 0.10.3).
• docs/BEST_PRACTICES.md follows all three, and its example scopes are valid paths.

Measurements

Measured on two release candidates of this code: 1c5acd42 (every check below) and 8fa30280, which only adds the planner instruction about denials (described under "Read-all answers"; its image was smoked, and the Hinglish questions were measured again on it). Only these notes changed after 8fa30280.

• Tests. The workspace suite (cargo test --workspace --all-targets) on Linux at umask 022, three runs of this code: 8,431 passed, 0 failed, 88 ignored (210 test binaries) twice; the third had one failure in a storage test of the dormant native subsystem, which this release does not touch, while the image smoke and the live Docker suite ran beside it on the same machine (the test passed 20 of 20 times alone, and its crate's suite 5 of 5). Every run includes the English regression gate (8 of 8, with the 64-combination cross-customer flag matrix) and no WAIT_TIMEOUT anywhere. Also green on 1c5acd42: rustfmt, the clippy ratchet (no new warning), doc tests, the script gates, the Python SDK, CLI, MCP (with a clean install) and TypeScript suites, the connectors, the server, backup and restore, retention, WAL-migration, usage-ledger, tenant-isolation, commitment and native-host smokes, and the multilingual mock evaluation with its top-5 diff and the kiraya check. The legacy-WAL restart smoke (not a CI gate) failed once under that load and passed 30 of 30 checks on a rerun; it fails the same way on 0.10.4.
• Image. Both candidates' multi-arch images boot on amd64 (ready in 1-2 s) and on arm64 under QEMU (6 s) with version 0.10.5, their own commit, the same 11 capabilities as 0.10.4 and these notes bundled; the multilingual and lifecycle smoke passed 42 of 42 and the cohort smoke 23 of 23 on each, and the live Docker default matrix 43 of 43 on each.
• Erasure on a copy of a 300-customer store (about 130,000 events, first boot of the copy): see the table under "Fixes". Both proofs were clean: 13 of 13 stores, every erased id answers 404, no file holds the canary's text or label.
• Read-all on the same copy: see the table under "Fixes".
• Single-customer answers do not change. The 115-question customer-journey set, replayed without the answer model in two request shapes: 113 of 115 contexts are byte-identical with two separate boots of v0.10.4. One of the other two also differs between those two v0.10.4 boots (a ranked cut-off that varies per boot); the other has the same size and one different event at a ranked cut-off, a per-build difference seen between builds of 0.10.3 as well (no code path this release changes is involved). One question per category, answered by the model: 12 of 12 correct.
• wait=indexed. About 51,600 ?wait=indexed writes against an earlier candidate of this release with the same wait code (sequential and concurrent writers on an otherwise idle server, and a concurrent write, seed and forget load): no 408, and the slowest write took 2.4 s.
• Model spend for all of this release's measurements on the store copy: $11.54.

Environment variables

docs/generated/env-vars.md has the details.

Error codes

On /v1/experience/bulk the code is on the failing item inside a 422 INVALID_ENVELOPE.

Startup warnings

• SETTINGS_OVERRIDE_CLEARS_SECRET: an entry of settings_overrides.json unsets an API key the launch environment sets (what an earlier console wrote when a card was saved with its key field empty). Re-enter the key in Settings, or delete the entry from the file.
• LABEL_LIMIT_INVALID: CORTEX_LABELS_MAX_COUNT or CORTEX_LABEL_MAX_BYTES is not a number or is 0; the default is used.

Rollback

A 0.10.4 image opens a data dir 0.10.5 has used: no column family, directory, WAL row kind, index field or key prefix is added, and the read-all result cache is in memory only. A Settings save that cleared a key writes the same file entry 0.10.4 writes for a removal. To roll back, stop 0.10.5 with the usual 120 s grace and start 0.10.4 on the same data dir. After a rollback, a Settings save with an empty key field from a client that sends the field removes the key again (the 0.10.4 behaviour), the label limits do not exist, and erasures on a store with expired retry records are slow again.

Checked on one data dir with the published v0.10.4 image and each release candidate, the cohort flags on, a graceful stop between each step (0.10.4 writes, 0.10.5 reads and writes, 0.10.4 again, 0.10.5 again): no column family, directory or on-disk format was added (the only new top-level entry, cohort_alias.key, comes from 0.10.4's cohort lanes), no error line and no panic, and every data check passed (84 of 85 checks; the remaining one expected the 404 a 0.10.3 rollback answers on /v1/cohort/query, where 0.10.4 answers 409 COHORT_DISABLED with its flags off, as it should).

Known issues

• A fresh read is still the map model's reading of each conversation. Reuse makes repeated answers over unchanged data agree; it does not make a reading deterministic. Two fresh reads of the same cohort can list a few different customers (in the release measurements, two fresh reads of the same question listed 113 and 112 customers). Reuse needs the same data revision, so on a scope that receives captures continuously most answers read afresh.
• A negated read-all question is read conversation by conversation. "Which customers did not discuss X?" lists the customers with at least one conversation that does not discuss X, as each conversation is read; it is not "customers none of whose conversations discuss X". For that set, ask the positive question and take the rest of the cohort.
• The planner's translation is a model's translation. A non-plain Hindi or Hinglish question reaches the readers as the planner's English translation; the server uses it only when it reads as English and keeps the question's negation and and, and otherwise passes the asker's own words, but it cannot check every other nuance. cohort.read.reader_question shows what the readers got.
• Two identical read-all answers at once read the cohort once. The second waits for the first (at most until 25 s before its own answer budget ends) and reuses its result. If the first stores nothing (a partial read, a result above 4 MiB, an answer cancelled late) and fewer than 10 s of read time are left after the wait, the second answers from the ranked sample with the note read: all waited for another answer ... instead of reading every conversation. Ask again for a full read.
• Label limits are not enforced by default (see "Label limits (announced)"). While enforced, a retry of a write the server accepted before enforcement was turned on, whose labels break a limit, answers 422 instead of replaying the earlier write: the limits are checked before the retry key is looked up. Turn enforcement on only after your clients' labels pass.
• A Settings key removed by an older console stays removed until it is re-entered (see "Before you upgrade").
• Everything listed under "Known issues" in RELEASE_NOTES_0.10.4.md still applies.

Clients

No client release. Every client published with 0.10.4 (Python SDK cortexdbai 0.12.0, TypeScript SDK cortexdbai 0.10.1, CLI cortexdb-cli 0.6.0, MCP cortexdb-mcp 0.8.0, connectors 0.2.22) works against 0.10.5 unchanged. In source, the Python SDK's experience(), the TypeScript ExperienceOptions.labels comment and the MCP server's memory_store tags parameter describe the announced label limits (docstrings only; no version change).

Notes

Single-node remains the only supported topology.