CortexDB 0.10.2 — indexed captures that do not queue behind disk syncs, one wait deadline, forgets that survive scope registrations, and the captured speaker on every recalled event
0.10.2 is a fix release on top of 0.10.1, the last published release. Everything in
RELEASE_NOTES_0.10.1.md still applies; this page lists only what changed since. It adds no
column family, directory or on-disk format, and nothing is migrated: a 0.10.1 server opens and
serves a data dir 0.10.2 has used (see "Rollback").
Summary
wait=indexedcaptures no longer queue behind other writes' disk syncs. The indexer and the bulk capture path write each batch to the log with one durable write instead of one disk sync per event. On a disk with ~3 ms syncs, a capture into a new scope during a 400-memory seed went from p99 1.1 s (max 1.4 s) to p99 0.28 s (max 0.46 s), and a 500-item/v1/experience/bulkfrom about 3 s to 0.25-0.5 s to acknowledge.- One wait deadline. A
?wait=that runs out answers408 WAIT_TIMEOUTafter about 30 s, not 60 s, says how long it waited and on what, and a bulk's budget now covers its inline indexing (the 408 then lists the batch's event ids). - Forgets and erasures are no longer refused by an unrelated scope registration. A capture
that registered a brand-new scope anywhere while a forget was being authorized made the forget
answer
403 POLICY_DENIEDwith nothing done. It now proceeds; a change inside its own footprint re-plans, and at worst answers a retriable409 AUTHORIZATION_STATE_CHANGED. - Recall reports who said each event. Several recall paths (long events found through a later
chunk, the holistic view's own-scope events,
prefer_exhaustive,order=recency) reported the capturing caller asobserved_actorand anunknownsubject. Every recalled event now carries the identity it was captured with, the same asGET /v1/events/{id}, and a cancelled capture leaves every recall view. GET /v1/events/{id}?scope=confines a by-id read to one exact scope (new capabilityevent_scope_filter_v1).budget_excerpt_v1is withheld whileCORTEX_RECALL_BUDGET_EXCERPTSis off, so a client that gates on it knows when excerpts are off.- A log line per request (
CORTEX_REQUEST_LOG, defaultslow), with the time spent in each stage, and the same request id on the audit rows a forget writes. - Fixes found after 0.10.1 shipped: embedding calls that asked for more than the client accepts in one response (a long document could stall indexing), empty descendant recall after a store grew past a size threshold while serving, the admin console's "degraded" label, and log noise from recall candidates a concurrent forget had just removed.
- No client changes. Every client released with 0.10.1 works against 0.10.2 unchanged (see "Clients" for what they will see differently).
Upgrade from 0.10.1
Before you upgrade: what clients and operators will notice
408 WAIT_TIMEOUTcomes at about 30 s, not about 60 s. A single?wait=indexedorconsolidatedcapture has one budget for both the indexer wait and the stage wait.detailsgainwaited_msandwaiting_on(indexerorstage). A client whose own timeout is a little over 30 s now gets the 408 instead of timing out itself.wait=consolidatednow has 30 s in total for indexing and consolidation, so a deployment with an enrichment delay or scope-quiet dispatch sees more 408s there. The budget starts after admission: a capture into a scope a running forget covers can first wait up toCORTEX_DESTRUCTIVE_LEASE_WAIT_SECS(default 120 s) to be admitted, as in 0.10.1.- A bulk
?wait=indexed|consolidatedcan now answer408while it is still indexing. Its one budget,min(30 s + 1 s per item, 600 s), now covers the inline indexing, which used to have no limit (such a batch waited and then answered 200 with its ids). The 408'sdetails.event_idslists the batch's events once they are written; poll them withGET /v1/experience/status?event_id=. - Resending after a 408 replays only what carried an
idempotency_key. The 408 messages now say so: an omitted key is minted by the server and never returned, so a keyless resend writes again. Send anidempotency_keyon every capture that waits, bulk items included. - The deprecated legacy dual-WAL backend (
CORTEX_UNIFIED_WAL=0) answers its own wait timeout with408 WAIT_TIMEOUTinstead of500 INTERNAL. 409 AUTHORIZATION_STATE_CHANGEDwhere a403or404used to come back. OnPOST /v1/forget,/v1/forget/preview,/v1/erasures,/v1/erasures/preview,GET /v1/erasures/{id}and its cancel, when scopes were registered or removed inside the request's footprint on each of 5 attempts. It is retriable:Retry-After: 1anddetails.retry_after_seconds. Nothing was changed; re-send the identical request. A client that treated the old spurious403as final now gets a retriable answer instead.- Recalled events carry their captured identity. On the paths listed above,
layers.events[].observed_actor,subject,scope_keyandwal_offsetchange from the capturing caller,{id: "unknown"}, the scope string and a per-partition sequence to the captured values, equal toGET /v1/events/{id}. So afilters.metadata.observed_actorrecall now keeps events captured on someone's behalf that it used to drop, and a client that comparedscope_keywithscopeon those items seesscope_keychange to the billing partition the event was captured under (for examplegoldencofor an event inorg:goldenco/user:alice).context_blockbytes do not change. - A cancelled capture leaves every recall view.
POST /v1/lifecycle/memory-event/{id}/cancelalready hid the event from granular recall andGET /v1/events; it now also leaves the default holistic pack,prefer_exhaustiveandorder=recency. - The Docker image sets
CORTEX_EMBEDDING_MAX_BATCH_ITEMS=256(it was 2048), and the OpenAI-compatible client clamps any value to what fits its 16 MiB response cap (340 texts at 1536 dimensions, 170 at 3072). A long document therefore makes more, smaller embedding calls. - The default request log writes a line for every response outside 2xx, unauthenticated
401probes included, plus every request slower than 1 s. A proxy or health checker that calls an authenticated route without a key writes one line per call. SetCORTEX_REQUEST_LOG=offto turn it off, or raiseCORTEX_REQUEST_LOG_SLOW_MS. - The image's
RUST_LOGaddscortex_embedding=warn, so the embedding clamp and retry warnings reach the container log. A deployment that sets its ownRUST_LOGkeeps it. - Forget-all latency under concurrent writes on a slow disk is about twice 0.10.1's (see "Performance"). With no concurrent writers, or on NVMe, it is unchanged.
- Holistic and
order=recencyrecall over large subtrees cost a little more: each event those paths add is now one point read of the captured event (measured +11% p50 for holistic recall at a parent of 3,000 events, +23% fororder=recencyover 2,000 events). Ranked granular recall is unchanged. capabilities[]lists 11 names:event_scope_filter_v1is new, andbudget_excerpt_v1is withheld whileCORTEX_RECALL_BUDGET_EXCERPTSis0,offorfalse.
Upgrade steps
Nothing is migrated and there is nothing to run afterwards. Replace the binary or image and restart as for any release, giving the server 120 s to stop (as in 0.10.1: it waits for running forgets and erasures and persists its snapshot). Then check:
GET /v1/admin/version:crate_version0.10.2, thegit_shayou deployed, andevent_scope_filter_v1incapabilities[].GET /v1/admin/ready:ready: true.- The log: if you set
CORTEX_EMBEDDING_MAX_BATCH_ITEMSyourself, aCORTEX_EMBEDDING_MAX_BATCH_ITEMS=<n> clamped to <m>warning tells you the value in effect. At 3072 dimensions the image's own 256 is above the bound, so the warning appears with no setting of yours; it is harmless. - If the request log is too verbose for your log pipeline, set
CORTEX_REQUEST_LOG=off.
Upgrading from v0.9.13
0.10.2 contains everything in 0.10.1. Read RELEASE_NOTES_0.10.1.md first ("Upgrade from
v0.9.13", "Plan the first start", "Upgrade steps"): its first start, background work and upgrade
steps are the same for 0.10.2, and this page's "Before you upgrade" applies on top.
Performance
How it was measured: one host, one server container at a time (4 CPUs, 3 GB of memory, mock embeddings, a fresh data dir), 0.10.1 and 0.10.2 alternated with the same client script. The disk matters: the gains come from fewer disk syncs, so they show on a disk whose sync takes milliseconds (here EBS gp3, 2.9 ms per 4 KiB synced write) and vanish on a fast one (NVMe, 0.13 ms). Your absolute numbers depend on your disk; the shape (latency that no longer grows with the size of the burst ahead of you) is what carries over.
Indexed captures into new scopes
Each round, four clients seed 400 memories into one scope while six clients capture with
wait=indexed into brand-new scopes, pausing 0.25 s between captures (the "no pause" row drops
the pause). In the rows "with forgets", a forget-all of the seeded scope runs during the seed and
another after it:
wait=indexed capture, EBS | 0.10.1 | 0.10.2 |
|---|---|---|
| p50 / p90 / p99 / max (ms), with forgets (10 min) | 66 / 546 / 1120 / 1403 | 72 / 156 / 276 / 460 |
| rounds whose p99 passed 1 s | 47 of 122 | 0 of 105 |
| captures over 500 ms / over 1 s, with forgets | 813 / 163 | 0 / 0 |
| p50 / p90 / p99 / max (ms), no forgets (6 min) | 94 / 1119 / 1569 / 1841 | 52 / 119 / 178 / 278 |
| captures over 1 s, no forgets | 394 | 0 |
| p50 / p90 / p99 / max (ms), with forgets, no pause (5 min) | 161 / 282 / 794 / 1372 | 159 / 276 / 371 / 992 |
| indexer cost per event in a page | 5.2 ms | 0.47 ms |
With real disk syncs, 0.10.2's tail (p99 178 ms, no forgets) is below what either build reaches
with disk syncs turned off (CORTEX_WAL_SYNC=0, a diagnostic setting that gives up durability:
p99 264 ms on 0.10.1, 294 ms on 0.10.2). On NVMe the two builds measure the same (p50 15 ms, p99
95 vs 93 ms).
Bulk captures
500-item /v1/experience/bulk, idle server, EBS, 6 requests | 0.10.1 | 0.10.2 |
|---|---|---|
no wait (ms) | 2953-3158 | 240-412 |
wait=indexed (ms) | 3157-3222 | 425-509 |
A bulk's log write is made while the indexer holds back every newer write, so the other clients'
wait=indexed captures used to wait out one disk sync per bulk item. Six new-scope
wait=indexed writers beside two clients sending bulks back to back (90 s per bulk size; the
first column is a 0.10.2 candidate that already had the indexer change but not yet the bulk one):
| Bulk size | Singles p50 / p99 (ms), indexer change only | Singles p50 / p99 (ms), 0.10.2 | Bulk log write while the indexer waits, p50 / max (ms) |
|---|---|---|---|
| none | 24 / 32 | 23 / 31 | - |
| 50 | 200 / 344 | 68 / 181 | 284 / 327 → 5 / 15 |
| 200 | 794 / 1534 | 284 / 439 | 719 / 1149 → 7 / 13 |
| 500 | 48 / 3715 | 54 / 994 | 2520 / 2795 → 12 / 15 |
What is left of the singles' tail is the bulk's own indexing (its embedding call and index
writes: about 1 s for 500 items here), because wait=indexed waits for the whole log (see "Known
issues").
Wait deadline
With an embedding provider that answers 503 to every call, a wait=indexed capture:
| 0.10.1 | 0.10.2 | |
|---|---|---|
| answer | 408 at 60.0 s, message "timed out after 30s" | 408 at 30.0 s, waited_ms: 30003, waiting_on: "indexer" |
A bulk wait=indexed fails fast with 502 INDEXING_FAILED on both.
Forget and erasure
Forgets and erasures sent while other clients registered new scopes:
| 0.10.1 | 0.10.2 | |
|---|---|---|
| forget-alls beside 12 clients registering new scopes in the same organisation | 39 of 52 refused 403 | 41 of 41 done at the first attempt (p50 374 ms) |
| forget-alls beside writers with no pause (arm above) | 26 403s; 10 of 70 never succeeded in 2 attempts | 50 of 50 done at the first attempt |
| forgets beside about 6.5 new child scopes per second inside the forgotten subtree | 283 of 283 | 287 of 287 at the first attempt |
POST /v1/erasures beside the registrations | 20 refused 403 | 67 accepted, 2 409 (retriable) |
GET /v1/erasures/{id} polls | 11 spurious 404s in 540 | 0 in 670 |
Forget-all latency (ms, answered 200) has a trade-off on a slow disk: the indexer no longer stalls, so a forget's search-index commit now competes with it for the index writer.
| Forget-all p50 / p90 / p99 / max (ms) | 0.10.1 | 0.10.2 |
|---|---|---|
| EBS, beside the seeding writers | 108 / 209 / 301 / 341 | 221 / 310 / 372 / 408 |
| EBS, beside writers with no pause | p50 115 | p50 299 |
| EBS, no concurrent writers | 151 / 210 / 224 / 382 | 155 / 227 / 258 / 383 |
| NVMe, beside the seeding writers | 50 / 69 / 86 / 102 | 51 / 79 / 96 / 108 |
Long documents and embedding calls
One 1,000,000-byte event (2,061 chunks) at 1536 dimensions, against an OpenAI-compatible endpoint:
| 0.10.1 (batch 2048) | 0.10.2 (image default 256) | 0.10.2 with 2048 set (clamped to 340) | |
|---|---|---|---|
| provider calls | 110, each response 64.6 MB and rejected | 9, largest response 8.1 MB | 7, largest response 10.7 MB |
| capture | 408 at 60 s; the indexer kept retrying | 200 in 1.6 s | 200 in 1.6 s |
What changed
Capture and indexing: one durable write per batch
A wait=indexed capture waits until the indexer has applied every write logged before it, in
every scope, not only its own event. The indexer applies the log in pages, and each page wrote its
derived rows with one disk sync per event: a 256-event page cost 256 syncs. On a disk with
millisecond syncs, a capture that landed behind a burst of writes (a seed, a connector backfill)
waited seconds for them. /v1/experience/bulk did the same with its captured events, while
holding back the indexer from every newer write.
Both now write a batch with one durable write: one write of the whole batch, then one sync, before anything is acknowledged. A crash keeps either the whole batch or none of it, and offsets are never reused or skipped. The stored rows are the same as before.
wait=indexed is still a wait for the whole log: a capture that lands behind a bulk waits for
that bulk's indexing, its embedding call included. Use wait=accepted and poll
GET /v1/experience/status when you do not need read-your-writes.
One wait deadline
- Single writes. The indexer wait and the stage wait after it share one budget of about 30 s.
A capture the indexer has not reached when it runs out answers at once; it used to wait a second
30 s for a stage it could no longer reach, and its message said 30 s.
details:event_id,target_stage,status_url,waited_ms,waiting_on(indexerorstage). - Bulk.
wait=indexed|consolidatedhas one budget ofmin(30 s + 1 s per item, 600 s)for the inline indexing and the stage barrier together. When it runs out during the indexing, the 408 (waiting_on: indexing,items) lists the batch's captured events indetails.event_ids(distinct, in item order, the list a 200 returns) as soon as the batch has been written; the indexing finishes in the background. During the stage barrier it iswaiting_on: stagewithacceptedand the sameevent_ids.wait=settledkeeps its settlement deadline; a bulk with nowaithas no deadline, as before. - Legacy dual WAL. The mirror's own wait timeout is a
408with{target_stage, waited_ms, waiting_on: "indexer"}, not a500. - What to do after a 408. The write is captured and durable. Poll
status_url(single) or each id inevent_ids(bulk). A resend replays, never duplicates, only when it carries the sameidempotency_key; a capture sent without one gets a server-minted key the client never sees, so resending it writes it again.
Forget and erasure: a scope registration no longer refuses a forget
A forget, an erasure and their previews check the caller's authorization once, then take a lease
on their footprint. The lease used to be refused whenever anything in the scope registry had
changed in between, and every capture that registers a brand-new scope, from any tenant, makes
such a change. The request then answered 403 POLICY_DENIED, with nothing done, and the same race
made GET /v1/erasures/{id} and its cancel answer a spurious 404.
The lease now compares the plan's own authorization boundaries, so a scope registered outside the
footprint does not matter. When a boundary appears or vanishes inside it, the request re-plans on
the current state (a caller the new boundary denies still gets 403, a job it cannot see still
404) and tries again, up to 5 attempts, then answers 409 AUTHORIZATION_STATE_CHANGED
(retriable, Retry-After: 1). Registry errors that are not a denial keep their own status instead
of a 403. Each refusal is logged at warn with its reason. An erasure refused this way marks its
job failed with nothing erased; a retry with the same idempotency_key restarts that job and
runs it, while a retry without one leaves a failed job record per refused attempt.
Diagnostics:
- The forget's info lines carry
elapsed_ms(monotonic, since the erasure began) andstep_ms; the "fragments erased" line addsbm25_publish_msandbm25_pending_adds_published(the pending search-index additions the forget's commit published). - The "erasure residual: pass complete" line adds per-step durations (
content_flush_ms,content_compact_ms,wal_ms,fragments_ms,fulltext_ms,fulltext_gc_ms,derived_flush_ms),fulltext_merge_error(with the segment) andmerge_vanished. A segment that a running search-index merge took, or that a delete emptied, is counted as vanished and carried to the next pass, not reported as a merge error. - The request log line of
/v1/forget,/v1/erasuresand their previews carrieslease_wait_ms,blocking_queue_msandengine_ms.
Recall: the captured identity for every event
Some recall paths built a layers.events item from an older internal record that holds only the
capturing caller, while GET /v1/events/{id} read the captured event:
- an event longer than
CORTEX_PARENT_HYDRATE_MAX_BYTES(6,144 bytes) found through a later chunk; - the default holistic view's events of the requested scope itself;
prefer_exhaustiveandorder=recency;- ancestor events, lineage, the evidence boost and artifact sources.
Those items reported the caller as observed_actor, {id: "unknown"} as subject, the scope as
scope_key and a per-partition sequence as wal_offset, and a filters.metadata.observed_actor
recall dropped them. Every item now carries the captured event's identity. A capture whose stored
copy is redacted is never served from the older record instead, and a cancelled capture now leaves
the holistic, exhaustive and recency packs too. context_block bytes are unchanged. This had been
wrong since v0.7.0.
Checked on a 13,080-byte event captured on behalf of another person and hit at deep chunks, under
granular and holistic recall, with prefer_exhaustive, order=recency, a 90-event holistic scope
and an observed_actor filter: every item's caller, observed_actor, subject, scope_key,
scope and wal_offset equal GET /v1/events/{id}. On 0.10.1 all eight checks fail. The
correct identity is also served for long events a 0.10.1 server wrote.
Recall: candidates dropped for want of content
A search-index hit carries no text; its text comes from the content store. A forget deletes the content first and publishes the search-index delete at the end, so a recall running meanwhile can find a hit with no content. It drops it (nothing is ever served) and used to warn "dropped candidates with missing or tenant-mismatched content". Now:
- a hit whose content an erasure removed during the recall, or shortly before, logs at debug;
- a hit with no content and no erasure behind it (an orphan index document) warns;
- a hit whose content belongs to another tenant warns.
GET /v1/admin/metrics counts them since process start under recall_empty_content_drops
(absent_during_erasure, absent_orphan, foreign_tenant).
Recall: after a store grows past the scope-map threshold while serving
A store whose log crossed 250,000 entries while serving, with no reader having built the scope-partition map while it was smaller, never got the map until the next restart, so the holistic view's descendant events came back empty (8 of 8 recalls over 35 s, found on a bulk re-ingest into a fresh data dir). The first check past the threshold now builds the map in the background (about 1 s for 266,000 entries); only the recalls in that window are degraded, and every later one serves the descendant events.
GET /v1/events/{id}?scope= (event_scope_filter_v1)
?scope=<scope> returns the event only when its own scope is exactly that scope. An event in a
sibling, ancestor or descendant scope is the same 404 NOT_FOUND as a missing id, byte for byte.
An empty, blank or malformed value is 422 INVALID_SCOPE_GRAMMAR, and a repeated scope
parameter is refused with a plain-text 400: never a read without the filter. Without the
parameter nothing changes; note that the static operator key and an auth-disabled server read
every scope of every tenant by id. A 0.10.1 server ignores the parameter, so send it only when
capabilities[] lists event_scope_filter_v1, or compare the returned scope yourself. It does
not apply to GET /v1/facts/{id}.
Capabilities
GET /v1/admin/version lists 11 names: 0.10.1's ten plus event_scope_filter_v1.
budget_excerpt_v1 is the only name an operator switch withdraws: it is not listed while
CORTEX_RECALL_BUDGET_EXCERPTS is 0, off or false, the setting that restores the 0.10.0
render, and the startup config lint logs RECALL_BUDGET_EXCERPTS_DISABLED. In 0.10.1 the name
stayed listed with the switch off, so a client that trusted it expected contexts inside their
budget and did not get them. Every other name stays listed whatever the switches say.
| Name | New in | What it tells a client |
|---|---|---|
event_scope_filter_v1 | 0.10.2 | GET /v1/events/{id} honours ?scope= (exact match) |
budget_excerpt_v1 | 0.10.1 | No context entry passes budgets.max_tokens; now absent while the switch is off |
| the other nine | 0.10.1 | Unchanged (RELEASE_NOTES_0.10.1.md, "Capabilities") |
Request log and request ids
CORTEX_REQUEST_LOG (off, slow or all; default slow) writes one info line per request
under the target cortex_api::request, which needs cortex_api=info in RUST_LOG (the image's
default has it):
request request_id=req_0197… method=POST route=/v1/experience status=408 total_ms=30009
admission_ms=4 lease_wait_ms=0 wal_append_ms=2 indexer_wait_ms=30000 queue_depth=412
outcome="timeout"
request_idis the id the response echoes inX-Request-Id(your own, when you send one);client_request_idappears when an error response minted another.routeis the route template, never the path;statusis 0 when the client went away first.- Stage timings appear when the stage ran:
admission_ms(the scope check and any auto-registration),lease_wait_ms(for a write, its admission time when a running forget covered its scope: an upper bound of the wait),wal_append_ms,indexer_wait_ms,queue_depth(the unindexed writes ahead of and including this one),stage_wait_ms, and for forgets and erasureslease_wait_ms,blocking_queue_msandengine_ms. outcomeisok,timeout(a wait ran out) orclient_gone.slowlogs requests of at leastCORTEX_REQUEST_LOG_SLOW_MS(default 1000), every status outside 2xx and every outcome other thanok;alllogs every request;offdoes no work.- No body, query string, path value or header other than
X-Request-Idis read.
The same id is now on the audit rows the request writes: the authentication layer's row and the
gdpr: true row of POST /v1/forget. A forget's audit trail, its log line and your own report
can be joined on it. (Before, the gdpr row always, and the authentication layer's row on a
successful request, carried a fresh id that appeared nowhere else.)
On the measured runs the line accounted for where a slow capture spent its time: in the
seeding test above, 0.10.2's slowest 1% of captures spent a median 245 ms in indexer_wait_ms
and 43 ms in admission_ms, and client and server times agreed within 2 ms at p50.
Embedding batches within the response cap
With CORTEX_EMBEDDING_MAX_BATCH_ITEMS=2048 (the 0.10.1 image default), one call for a long
document asked for more than the embedding client accepts in one response (a 630-chunk event at
1536 dimensions is about 21 MB of formatted numbers against a 16 MiB cap). The call failed and was
retried unchanged, and indexing stalled on that event. Now:
- the OpenAI-compatible client reports the most texts per call whose response fits the cap,
floor(12 MiB / (24 × dimensions + 128)): 340 at 1536 dimensions, 170 at 3072, 2006 at 256; - the configured value is clamped to it on every flush, and a value set above it logs one warning
at startup (
CORTEX_EMBEDDING_MAX_BATCH_ITEMS=<n> clamped to <m> ..., targetcortex_embedding); - an over-cap response reads as "too large", so the batch is split in half instead of re-sent;
- the Docker image sets 256.
Admin console
The readiness pill said "degraded (mock embeddings)" for every degraded state, including a failing
embedding provider (for example an expired key answering 401). It now names the reason (mock
embeddings, or embedding calls failing) and shows the server's detail on hover.
API contract text
/v1/openapi.json descriptions are rewritten for API users (no internal project codes, ticket ids
or storage internals). POST /v1/forget now references ForgetRequest like its preview does (only
scope is required; reason is an alias of audit_note; from_preview_id is documented; unknown
fields are refused, as the server always did), which tightens clients generated from the spec.
POST /v1/answer marks question_type deprecated and ignored and documents skip_answer_llm,
conversation_id, view, order, budgets and include. The new 408s and 409s are declared,
with Retry-After on the 409s, and GET /v1/events/{id} documents ?scope=.
Environment variables
| Variable | Change |
|---|---|
CORTEX_REQUEST_LOG | New. off, slow (default) or all; see "Request log and request ids". |
CORTEX_REQUEST_LOG_SLOW_MS | New. The slow threshold, default 1000 ms. |
CORTEX_EMBEDDING_MAX_BATCH_ITEMS | The image default is 256 (was 2048; the binary's own default stays 2048); the OpenAI-compatible client clamps it to its response bound. |
CORTEX_RECALL_BUDGET_EXCERPTS | 0/off/false now also withholds budget_excerpt_v1 and logs a startup lint. |
RUST_LOG | The image default adds cortex_embedding=warn (so does the binary's built-in default). |
docs/generated/env-vars.md has the full rows.
Error codes
| HTTP | Code | Where | Change | Retriable |
|---|---|---|---|---|
| 408 | WAIT_TIMEOUT | POST /v1/experience, /v1/experience/bulk with ?wait= | One budget per request (about 30 s single; min(30 s + 1 s per item, 600 s) bulk, inline indexing included). details.waited_ms, details.waiting_on; bulk adds event_ids. The legacy dual-WAL mirror's wait timeout is a 408, not a 500. | yes: poll, or resend with the same keys |
| 409 | AUTHORIZATION_STATE_CHANGED | /v1/forget, /v1/forget/preview, /v1/erasures, /v1/erasures/preview, GET /v1/erasures/{id} and its cancel | New on these routes. Replaces a spurious 403 POLICY_DENIED or 404. Retry-After: 1, details.retry_after_seconds. (On /v1/recall and /v1/answer the same code remains a 503.) | yes |
| 422 | INVALID_SCOPE_GRAMMAR | GET /v1/events/{id} | New there, for an empty or malformed ?scope=. | no |
Startup warnings
RECALL_BUDGET_EXCERPTS_DISABLED:CORTEX_RECALL_BUDGET_EXCERPTSis off, so recall renders as 0.10.0 did andbudget_excerpt_v1is not listed.CORTEX_EMBEDDING_MAX_BATCH_ITEMS=<n> clamped to <m>(targetcortex_embedding): the value in effect is below the one configured.
Rollback
0.10.2 adds no column family, directory or on-disk format; its batched log writes store the same
rows a one-by-one write stores. To roll back, stop 0.10.2 with the usual 120 s grace and start the
0.10.1 build on the same data dir. A rollback brings back 0.10.1's behaviour, including the issues
this release fixes (and, on the 0.10.1 image, its embedding batch default of 2048). A client that
gates on capabilities[] stops sending ?scope= by itself; one that does not keeps sending it,
and 0.10.1 ignores it.
What was checked, on one data dir, with a graceful stop between each step:
- 0.10.1 wrote English, Devanagari and Hinglish captures, a 32 KB turn, a 13 KB event captured on someone else's behalf and a 200-item bulk, and ran a forget-all and a forget by id.
- 0.10.2 read all of it (a budget excerpt,
?scope=giving 200 for the event's own scope and 404 for an ancestor, and the correct identity for the long event 0.10.1 wrote, which 0.10.1 itself had served with the caller's identity), wrote single captures, a 300-item bulk and an 11-chunk Hindi document, and ran a forget-all and a forget by id. - 0.10.1 again opened the store and served everything 0.10.2 had written (the bulk samples, the long document by id); what 0.10.2 had forgotten stayed gone. It wrote, bulk-wrote 100 items and forgot the document by id.
- 0.10.2 again read everything, including the writes made during the rollback, and the
forgotten items answered
404.
Every phase stopped cleanly (exit 0) and logged no error. After each phase the column families of every RocksDB store (from its latest options file) and the data dir's top-level entries were listed: 0.10.2 added none, before or after the rollback. The final 0.10.2 build passed all 72 checks.
A rollback from 0.10.2 straight to v0.9.13 was not re-run. 0.10.2 stores nothing 0.10.1 does not,
so RELEASE_NOTES_0.10.1.md, "Rollback", describes it, with its limits.
Known issues
New or changed in 0.10.2:
- A storm of new scopes inside the forgotten subtree still blocks a forget. Under about 28
new child scopes per second inside the footprint, every re-plan found the footprint moved again:
6 of 6 forgets exhausted 10 client attempts with
409 AUTHORIZATION_STATE_CHANGEDfor as long as the storm lasted. 0.10.1 answered403there, so neither makes progress; about 6.5 per second converged at the first attempt (287 of 287). Pause the captures into the subtree, or retry after them. wait=indexedstill waits for the whole log, not for the capture's own event: a capture that lands behind a large bulk waits for that bulk's indexing (about 1 s for a 500-item bulk in the measurement above), so its tail still grows with the size of concurrent bulks.- A bulk is slow beside a stream of new-scope captures. A 500-item bulk into one existing scope took 7-9 s beside six clients registering new scopes (0.25-0.41 s alone), on 0.10.2 before and after its bulk change. Its admission, log write and indexing account for about 1 s of that; the likely cause, not yet confirmed, is the bulk's per-item scope check after admission, which queues behind the registry rewrite each new scope makes (next item).
- New-scope capture admission grows with the scope registry: each new scope rewrites the registry snapshot. In a 10-minute test admission p50 grew from 13 to 45 ms (from 31 to 114 ms with writers that never paused), the largest growing term once the disk-sync stall is gone. 0.10.1 has the same cost, hidden behind the stall.
- Other scope-checked routes can still be refused by a concurrent registration.
POST /v1/lifecycle/memory-event/{id}/cancel(a spurious404), the/v1/conflictsroutes (403or404),GET /v1/usageandPOST /v1/understanding/synthesizestill check against the registry state they planned on, so a new scope registered anywhere at that moment refuses them with nothing done. Retry. - On
/v1/scopesmutations the409 AUTHORIZATION_STATE_CHANGEDand409 SCOPE_STATE_CHANGEDbodies sayretriable: false. Nothing was applied; re-sending the mutation is safe. - The request log times the response headers.
total_msends when the headers are sent, so a streamed body (the lifecycle stream, a streamed answer) is logged when it starts, and a client that leaves during the body is still loggedok. - More search-index merge warnings. The faster indexer commits more often, so the harmless Tantivy warning "couldn't find segment in SegmentManager" appears more often (87 against 11 in one 10-minute run). No data or latency effect.
- The deprecated legacy dual-WAL backend (
CORTEX_UNIFIED_WAL=0) restarts slowly after repeated hard kills with large events in flight (boot times of about 1 s, 15 s, 40 s, then over 150 s in one test), and an acknowledged capture was not yet listed 120 s after one of them. 0.10.1 behaves the same. Use the default unified WAL.
Carried over from 0.10.1 (still present in 0.10.2):
POST /v1/experience?wait=settledcannot succeed: the capture is written, then the request answers503 SETTLEMENT_UNAVAILABLE. A retry without the sameidempotency_keywrites it again. Usewait=indexed.- The
lifecycle_streamlink a bulk capture (?batch_id=) or an erasure (?erasure_id=) returns is not filtered by that id: the stream carries every lifecycle event the caller may see. PollGET /v1/experience/statusorGET /v1/erasures/{id}instead. GET /v1/facts?subject=compares the stored subject, which for extracted facts is the extracted name, not theent_id the response shows, so filtering by a returned id finds nothing;objectis not implemented.- A per-request
answer_modelon/v1/answeris echoed in the diagnostics but not applied. CORTEX_VERIFIER_URL=(empty) does not turn the verifier off: it falls back to the OpenAI URL. To keep the verifier off, leaveCORTEX_VERIFIER_API_KEYandOPENAI_API_KEYunset.GET /v1/conflicts/{id}and its resolve answer an unknown id with404 NOT_FOUND, not the documentedCONFLICT_NOT_FOUND.- A request body the server cannot parse (for example an unknown field on
/v1/forget) is refused with a plain-text422, not the JSON error envelope. - Everything under "Known issues" in
RELEASE_NOTES_0.10.1.md.
Clients
No client changes: no client version is bumped, and cortexdbai 0.12.0 (PyPI) and 0.10.0 (npm),
cortexdb-mcp 0.8.0, cortexdb-cli 0.6.0 and cortexdb-connectors 0.2.22 work against 0.10.2
unchanged. Every change is additive or on an error path. What clients will see:
- Python and TypeScript SDKs (default timeout 60 s): a wait that runs out now arrives as a
V1APIErrorwithstatus408 anderror_codeWAIT_TIMEOUTat about 30 s, withdetails.status_urlordetails.event_ids, instead of a client-side timeout at 60 s. The new409 AUTHORIZATION_STATE_CHANGEDon forget and erasure arrives as a plainV1APIErrorwithretriabletrue, not as theV1PolicyDeniedErrorthe old spurious403raised: retry it afterRetry-After. - Neither SDK has a method for
GET /v1/events/{id}, so?scope=has no client method yet; call the route directly. cortexdb-cli: its HTTP timeout is 30 s, the same as the server's wait budget, so--wait indexedcan still end in the CLI's own timeout rather than the408. The capture is durable either way; check it withGET /v1/experience/status.cortexdb-mcp: its get-by-id tool readsGET /v1/events/{id}without?scope=.cortexdb-connectors: a forget that meets the registration race now gets a retriable409instead of a403; the connector's batch stays durable for its next retry, as before.- Framework adapters (
cortexdbaiintegrations): their forget tools run a scope-wide forget (cascade: redact_events,confirm_all: true) and ignore the query they take. This is not new in 0.10.2; do not expose those tools to an agent unless a scope-wide forget is what you want.
Notes
Single-node remains the only supported topology.
The single-capture, forget and registration figures above were measured on a 0.10.2 candidate before its last change, which touched only the bulk capture write and the request ids; the bulk figures, the upgrade and rollback check and the smoke tests were run on the final build. The recall-cost figures (+11% and +23%) come from the change's own measurement and were not re-run on the final build.