Code memory (the Code Intelligence Plane)

CortexDB indexes source repositories into a typed, versioned code graph and serves it with honest precision labels. This page is the operator + API reference for the feature. What a server may register, and what it indexes, is under Registration safety.

Quickstart

# 1. turn the feature on (admin UI → Features → "Code memory", or:)
export CORTEX_CODE_PLANE=1                     # the plane and /v1/code/*
export CORTEX_CODE_BUNDLE=1                    # optional: also feed recall
./target/release/cortexdb                      # restart to pick it up

# 2. register a checkout (indexes synchronously, reports fact counts)
cortexdb code add myrepo /path/to/checkout --scope org:acme/team:eng

# 3. ask it things
cortexdb code status myrepo
curl -H "Authorization: Bearer $TOKEN" \
  "localhost:3141/v1/code/symbol/resolve?repo=myrepo&name=settle"

Off is free: with CORTEX_CODE_PLANE unset nothing is constructed — no stores are opened, no watchers run, /v1/code/* answers 503 CODE_PLANE_DISABLED, and recall behaves exactly as before.

CORTEX_CODE_BUNDLE is a second, independent switch. It attaches the plane to /v1/recall as a code evidence lane: recall then also returns layers.code and a trailing ## Code section, inside its own 25% slice of the token budget. The lane serves only repositories the caller may read: bound under the request scope, ingested by the caller's own tenant, with the Code Plane's own read capability (diagnostics.read) on their bound scope, readable there the way an event would be (scope.read.local and a Reader seat), and — for a verified per-actor caller — governed by a scope roster, as on /v1/code/*. A token kept off /v1/code/* therefore gets no source through recall either. A repository left out for that reason is never named in the diagnostics. The same rule is applied again, region by region, whenever a pack that carries code is reused (use_pack_id on /v1/answer and /v1/compose, and every stream fence): a caller who may not read one of its repositories is refused the whole pack with 403 POLICY_DENIED, since its context_block quotes the same source. With the plane on and the bundle off, /v1/code/* works and recall is byte-identical to a server that has no code plane at all.

What it knows, and how sure it is

Every fact carries a precision tier and a resolution status, and both travel to the answer:

There is no global name-match fallback anywhere. A file whose name matches your query but which no resolution rule reaches is never linked (tests/never_guess.rs is the permanent regression).

The syntax tier also stores named runtime/framework resources as typed edges: routes, configuration reads, callback registrations, event handlers and emissions, React/reducer state transitions, and literal outbound HTTP endpoints. These remain labeled Dynamic; querying a resource walks the edge back to its owning handler and file without pretending that syntax is compiler proof.

HTTP API (/v1/code/*)

Bearer auth; reads need diagnostics.read, mutations need admin.compact. Every repository is bound to a scope path at registration (below), and both the capability and the caller's roster role are checked against that bound scope, with the token's own scope confinement applied on top. Any denial — unknown name, another tenant's repository, a missing capability, a token confined elsewhere, no roster role — is the same 404 REPO_NOT_FOUND, and the row is absent from listings: existence never leaks.

A repository no roster governs is never served to a verified caller. The scope registry grants every role on a path no record governs (the scratchpad tier), which is the right default for a scratch memory and the wrong one for source code. So for a verified per-actor token, a repository whose bound scope has no registered scope at or above it answers the same 404 REPO_NOT_FOUND and is left out of listings. Static-key and auth-disabled operator identities skip rosters everywhere and are unaffected. A roster that cannot answer is a 503, for a listing too — never a silently shorter list.

Registration always leaves the parent governed. POST /v1/code/repos needs admin.compact and, for a verified caller, Owner on the parent scope. When no roster governs the parent yet, registration registers it with the caller as its Owner — exactly the admission a first memory write to that scope runs, under the same scope.auto_register and allowed_scope_types settings, and only for a caller that also holds what that memory write needs, scope.write on the parent (where a token's scopes confinement applies) — so the repository and any memory under that scope share one roster. A parent that another actor governs by the time the registration reaches it is decided by that roster, never provisioned over. Consequences worth knowing:

• A root is never claimed. A request that names no scope binds under the tenant root org:<billing id>; if that root is not registered, the registration is refused with 403 SCOPE_REGISTRATION_REQUIRED. Pass a team scope ("scope": "org:acme/team:eng"), or register the root first (POST /v1/scopes) and seat its members. Claiming the root would make one agent Owner of the whole tenant.
• The first registrant claims an unregistered team scope, as the first memory writer does. A deployment that wants team isolation from day one registers its team scopes (POST /v1/scopes) and seats their members before anyone writes.
• Another team's registration into a scope it does not own is 403 POLICY_DENIED, and creates nothing. So is a registration that would provision a parent the token's scope.write does not reach.
• Every refusal above, and one under scope.auto_register: false or allowed_scope_types, answers before the path is read, so it is the same whether the path exists or not.
• The path is checked before the parent is provisioned: a request refused with 422 REPO_PATH_NOT_ALLOWED, or whose path does not exist, leaves no scope registered. If indexing fails after the parent was provisioned, the repository is not registered but the parent scope stays registered, as after a failed memory write.

The bound scope is <parent>/repo:<name>-<12 hex>, where the suffix is a digest of the canonical checkout path. The leaf comes from the repository name, so two names registered from one checkout bind to two scopes. Names are [A-Za-z0-9_-], at most 115 characters (400 INVALID_REPO_NAME otherwise), which keeps the leaf inside one scope id untrimmed.

Errors use the standard v1 envelope ({"error_code", "message", "retriable", "request_id", "details"} — docs/schemas/openapi.json#/components/schemas/ErrorEnvelope), the same one every other v1 route emits, so both SDKs raise a typed error with a real error_code. Deep traversals without an explicit bounded plan are refused (TRAVERSAL_REFUSED), not silently truncated.

The code.context envelope

{ "envelope": {
  "items": [{ "path": "core/billing.ts",
              "citation": "myrepo:core/billing.ts@9e49cb54ea4b1262#L42-L61",
              "snippet": "export function settle(…) …",
              "reason": "changed | impacted-caller | lexical-anchor",
              "truncated": false }],
  "files_considered": 7, "files_included": 4,
  "token_budget": 2000, "tokens_used": 1180,
  "uncertainty": ["content unavailable: gen/schema.ts"],
  "warnings":    ["prior attempt on `settle`: broke invoice rounding tests"] },
  "snapshot": "3f1c9a…",   // base commit of the indexed checkout
  "generation": 7 }

A citation is repo:path@<snippet-hash>#Lstart-Lend. <snippet-hash> is the first 16 hex characters of BLAKE3 over the exact snippet text returned, so you can verify a citation by hashing its snippet. It is not a git commit, blob or tree id (git show <hash> does not work), and it is not the file's content-store key either: it pins the served bytes. snapshot is the commit the indexed checkout is based on (HEAD when it was last indexed; worktree for a checkout with no Git history), and generation is the index generation the bytes were served from. The plane indexes the working tree, uncommitted edits included, so for a file edited since that commit the snippet does not exist at snapshot (git show <snapshot>:<path> differs): the citation hash still verifies against the snippet, and generation is what identifies the served state.

Priorities are structural, not scored: changed files → their impacted callers → fused symbol, graph, semantic, and lexical anchors. Each file is served as a verbatim query-centered window capped at 2 KiB, with exact line ranges in its citation, so one large file cannot consume the whole envelope. The token budget is a hard ceiling (property-tested over random inputs); every excerpt, missing file and unindexed path appears in uncertainty.

A snippet is served only from a file the published generation indexed, and only as the bytes it indexed. A changed_paths entry naming anything else — a credential file, .git/config, a file .gitignore excludes, any other file the scan did not admit, or a file edited since the generation was built — is never read for the answer: it appears in uncertainty as content unavailable: <path>.

workspace/context accepts repos (empty means every repository visible to the authenticated tenant), a task, optional changed_paths keyed by repo, and one hard token budget. It divides that ceiling deterministically across at most 16 repositories and returns repo-qualified items plus the exact snapshot and generation consulted for each repository. Missing or cross-tenant repositories return the same 404; repository existence never leaks.

graph/query accepts up to 16 exact indexed roots and 16 predicate names, direction (outgoing, incoming, or both), and explicit depth/node/edge budgets. Depth is capped at 6, nodes at 5,000, and edges at 10,000. Results carry predicate, tier and status on every edge plus the exact generation and manifest consulted. Dynamic edges are excluded unless include_dynamic is explicitly true. This provides general graph exploration without exposing an unbounded database query language to an agent.

Roots are exact indexed names, and a bare name answers with EVERY identity that carries it (ambiguity is returned, never collapsed). A file root (for example os-test.cc) also roots every definition the file defines, unless defines is among the predicates: calls live in function bodies, so roots=["os-test.cc"] with the default calls answers with the calls its functions and test bodies make, in every language, plus any the file makes at module level. The named roots are walked first and return every edge they return alone, whatever the file's size; the definitions are walked after them, as roots, with the node and edge budget that leaves, and a definition joins roots and nodes only with an edge of its own, so one without any costs no budget. A definition a named root already reached is still walked as a root, so with budget to spare the answer is the one naming every definition would give: each edge appears once, at its smallest depth from any root. file_definition_roots counts the definitions that joined; at most 1,024 per query are considered, and truncated is true when there were more or the budget ran out first. A query only for edges no definition has — outgoing imports, exports, extends — does not walk the definitions at all, so their number never marks its answer truncated. Asking for defines keeps the plain file node, so file --defines--> definition edges stay inspectable one hop at a time.

C++ members also answer to their qualified names: Class::method and ns::Class::method (up to eight enclosing scopes), for members defined in the class body and out of line alike, so buffered_file::close resolves to exactly that member while close still returns every close. Every node carries label (the last name) and qualified_name (for C/C++ members, the longest qualified spelling, itself a valid root; otherwise null). Qualified roots are C++ only: dotted Class.method forms for other languages would collide with dotted resource names such as event invoice.created. A class declared with an export macro between class and its name (class FMT_API file {, how fmt declares its exported classes) is still the class file: the grammar reads it as a function returning class FMT_API, and the extractor recovers the name (skipping final, before any base clause) and qualifies the members inside it, so file::descriptor resolves. A member declaration the grammar reads as a call there (~file() noexcept;, file(const file&) = delete;, directly or in an #if/#ifdef branch) is no call: the class calls nothing it declares. Recovery is best effort: a member the grammar loses inside such a class — for example a definition directly after an access label — stays unindexed, and a macro before class (EXPORT class X {) or a macro class with several base classes is not recovered. Attribute forms (__declspec(…), __attribute__((…)), alignas(…), [[…]]) parse as ordinary classes. Qualifiers opened by a macro (FMT_BEGIN_NAMESPACE) are invisible at syntax tier.

C and C++ extraction follows the definition through pointer, reference and parenthesized declarators (redisContext *redisConnect(…), T &f()), so calls in those bodies belong to the function, not the file. Destructors (~X, in-class or X::~X) and operators are definitions. A gtest body — TEST, TEST_F, TEST_P, TYPED_TEST, TYPED_TEST_P or GTEST_TEST — is a definition named Suite.Name (the --gtest_filter spelling), so a function exercised only by tests has callers. Qualified and template callees (fmt::report_windows_error(…), f<T>()) are extracted at name precision like every syntax-tier call. A .h file is parsed as C++ when its code (comments and literals excluded) contains a construct only C++ has — template <, namespace … {, class X {, an access specifier, std::, an extension-less #include <vector>, using namespace, or a struct with a base clause — and as C otherwise; the choice depends only on the file's bytes. Catch2, doctest and Boost.Test cases are not named yet.

Set output (on graph/query or graph/export) to json, graphml, neo4j_cypher, vis_html, d3_tree_html, svg, obsidian_vault, markdown_wiki, or dot. Each artifact is deterministic and generated from the exact same tenant-scoped, generation-pinned, budgeted slice. The Obsidian format is a directly extractable ZIP vault; HTML outputs are interactive and self-contained (no CDN or JavaScript package dependency); GraphML, Cypher and DOT are tool-importable. Whole-repository browser renders remain intentionally bounded—large graphs should be explored as slices.

Certificates

{ "certified": { "plan": "count-call-sites(…) = compiler-tier reference occurrences",
                 "repo": "myrepo", "snapshot": "abc123…",
                 "manifest_hash": "…",           // the epoch observation
                 "precision_requirement": "compiler",
                 "input_blocks": ["…"],          // never empty
                 "coverage": { "compiler_tier_pct": 100.0, … },
                 "answer": { "Count": 3 },
                 "scope_paths": ["src"],          // omitted for the whole repository
                 "scope_languages": ["c-family"],
                 "payload_hash": "…" } }

The scope is the pinned manifest, not a wish. POST /v1/code/certify reads refs/heads/main and derives its evidence from that manifest, in one place (cortex-code-index::certify_scope), under three definitions that exist nowhere else:

• in scope — a manifest entry that is code a compiler-tier index can cover. Every excluded entry is counted in out_of_scope_files and broken down in excluded, never silently dropped:
  • not_code — no grammar with a query pack: docs, data, config and lockfiles (markdown, json and yaml have parse-only grammars and are still not code);
  • other_language — code in a language family no imported index covers and that is not the subject's own family, once one is covered: a scip-clang index can never cover a shell script, so a C repository's .sh files no longer block its certificates. The families are conservative interop groups — C/C++/Objective-C, TypeScript/JavaScript, and the JVM languages; every other language is its own family. The subject's own family always stays in scope — it is where the subject's call sites are — so asking about a Python function with only a TypeScript index imported answers incomplete, naming the Python files. The family comes from the file an imported index defines the subject in, a provisional id's path, or a SCIP id's indexer (cxx, scip-typescript, scip-java/semanticdb, scip-python, rust-analyzer, scip-go); a subject none of these place excludes no family. With nothing imported, every code file stays in scope, so the refusal names each file that needs an index;
  • outside_paths — outside the optional request paths: repo-relative prefixes matched on segment boundaries (src covers src/a.c, never srcs/a.c), at most 64. paths lets a repository whose examples or vendored adapters cannot be indexed certify the part that is. The certificate records the filter (scope_paths) and the families (scope_languages), both inside the payload hash;
• compiler-tier — a block whose extractor id starts with scip/ and whose file hash equals the manifest entry's content hash. SCIP left over from an earlier revision of a changed file is not coverage;
• coverage — the published ledger's summary, with compiler_tier_pct = 100 × files carrying a compiler-tier block ÷ in-scope files. Every certificate and every incomplete carries it.

An empty scope is refused: a repository with no compiler-tier blocks answers incomplete with "no compiler-tier scope: import a SCIP index first" rather than certifying Count(0) over nothing. The rule lives in the kernel, not in the route, so no caller can construct a vacuous certificate. The request refuses unknown fields (422) rather than ignoring them, and a malformed paths entry is 400 INVALID_PATHS. count-call-sites counts distinct reference occurrences: a re-import after an importer upgrade leaves two compiler-tier blocks for one file version, and an occurrence in both is one call site.

Certificates are hash-verified, never signed: verify re-loads every input block by identity and recomputes the answer, so tampering with the answer, the block bytes, or the input list all fail. Any file in scope without compiler-tier coverage, and any unresolved or ambiguous row touching the subject, yields {"incomplete":{"reasons":[…],"coverage":{…}, "scope_paths":[…],"scope_languages":[…],"out_of_scope_files":n, "excluded":{"not_code":n,"other_language":n,"outside_paths":n}}} — a wrong certificate is worse than none. GET /v1/code/coverage/status reports the same scope_languages, out_of_scope_files and excluded for the whole repository. A concurrent write that moves the snapshot head during certification yields incomplete too (retry-once rule); a background refresh that moves the repository's generation while the answer is computed is a 409 CODE_INDEX_REFRESHING instead — the read was of an index that is no longer the one being served.

Compiler-tier blocks are durable. POST /v1/code/import/scip stores each accepted block in the repository's content store (a semantic_by_file namespace keyed by file content hash) before materializing it, attaches it to its manifest entry and republishes the head. They therefore survive a restart, an incremental refresh and a full rebuild — re-attachment is by file content identity, so editing one file never drops another file's coverage. Blocks no manifest reaches are collected by POST /v1/code/gc, which also sweeps semantic_by_file entries whose blocks are all gone (deleted_semantic_index_entries in the report).

Configuration

cortexdb config-lint lists the primary operational knobs with their definition sites. Every one of them is translated once, in cortexdb/src/main.rs::code_plane_from_env, into typed options; neither the plane nor the cortex-code-index kernel beneath it reads the environment, so what config-lint prints is what the plane runs on. There are no separate vector, rank or auto-SCIP toggles: the semantic channel is switched by CORTEX_CODE_EMBEDDING_PROVIDER alone (unset = no code model, no API cost, no vector index), the rank prior is library API with no ambient switch (D-04's ablation measured no gain), and compiler-tier import stays explicit (D-11).

Semantic retrieval is opt-in and additive. It builds one bounded structural representation per file from its path, definitions, documentation/comments, and frequent identifiers for code-bearing query-pack files. Parse-only documentation/data remains universally available through BM25 without paying code-model inference cost. Embeddings are reused by representation hash and stored in a TQ2-compressed HNSW index; provider failure falls back to exact-path, graph, and BM25 channels. The same persistent vectors can re-score the already admitted candidate tail. Rank one is always retained; ranks two and three are protected only when explicit or corroborated near the head of independent evidence channels. This uses one cached query embedding and bounded point reads, never query-time document embedding. Live vector-to-path mappings are reconstructed from the pinned manifest on restart and changed files update incrementally.

Bulk indexing and interactive queries may share one endpoint, but long local model batches can then create severe query-tail latency. Set CORTEX_CODE_QUERY_EMBEDDING_URL to separately provisioned capacity when backfills and interactive traffic overlap. The query service uses the same model and dimensions, so its vectors remain compatible with the persistent index; only the query instruction and credential may differ.

Compiler-tier precision (SCIP in CI)

Syntax + manifest tiers need nothing. For compiler-grade answers, run your language's indexer in CI and upload the result:

- run: npx @sourcegraph/scip-typescript index --output index.scip
- run: cortexdb code import-scip myrepo index.scip

Verified indexers (vendored fixtures + passing import tests): scip-go 0.2.7, @sourcegraph/scip-typescript 0.4.0. Others are expected to work — the importer validates per-document against the snapshot and reports exactly what it accepted and rejected — but only the two above carry compiler_precise: true in the matrix, because only they have been measured.

The import derives compiler-tier call edges: a reference to a callable (a SCIP method descriptor, which is how every indexer spells functions and methods) inside a definition's body becomes a calls row, Compiler tier, Exact status, stored in the imported block itself. graph/query and impact traverse those edges next to the syntax-tier ones, and report tier: "Compiler" / weakest_tier: "Compiler" for them; because the edges live in the durable block, they survive restarts and rebuilds and leave with the block when the file changes. References to types, fields and macros stay references only. A definition's body is its enclosing_range (the typed or the deprecated form). scip-clang 0.4.0 emits none, so a C/C++ function takes its body from the syntax tier of the same pinned file version — the unique same-named syntax function whose span contains the definition line — and gets no body (and no edges) when there is not exactly one. A class, struct, field or macro gets no body at all: a class body holds member declarations (void close();), which are not calls, so a class is never a caller. The name span is never used as a body. The response reports derived_calls and definitions_without_body — the functions and methods left without a body; other definitions never have one and are not counted. Compiler-tier call and reference targets are reachable by name whatever the indexer's scheme (cxx, rust-analyzer, scip-*). Linking syntax and compiler identities (the alias table, T-402) is not wired yet: a name resolves to both identities, each with its own edges.

Stats: semantic_blocks counts attached SCIP blocks; semantic_chunks* are the optional embedding index (CORTEX_CODE_EMBEDDING_PROVIDER), which SCIP does not feed.

Coverage, GC, and portability

• Coverage: POST /v1/code/import/coverage parses lcov and records it on the repository — {covered_files, covered_lines, recorded: true} — persisted with the runtime state, so a restart cannot silently lose it. tests.for_change then labels a row observed (a recorded run executed that definition) instead of static (a call path exists). Absent coverage is disclosed, never upgraded; an lcov payload that parses to nothing is refused with 422 LCOV_EMPTY rather than recorded as empty. observed rows are bounded to the files the QUERIED symbols live in: the manifest walk that produces definition spans is a CAS read per block, and a question about one symbol must not answer with coverage for every covered definition in the repository. A repository with no published snapshot keeps serving its static rows and says why in notes.
• GC: two-phase (mark, then delete on the next cycle if still unreachable), with pins, overlay TTL (CORTEX_CODE_OVERLAY_TTL_DAYS) and per-repo quotas. Every destructive step appears in the report, including deleted_semantic_index_entries — semantic_by_file entries whose blocks are all gone.
• Portability: export-index / import-index move a whole index between servers. The bundle is tree-rooted and every entry's identity is recomputed on import before any write — a single flipped byte rejects the whole bundle and leaves the store untouched.

Registration is a create, not a rebind

POST /v1/code/repos binds a NEW name. A name this tenant already has is refused with 409 REPO_EXISTS: the plane cannot re-open stores the live runtime holds (RocksDB takes an exclusive lock on the repository directory), and silently returning the existing repository would hide the path or scope the caller asked for. Remove it first, then register it again. A journaled repository the last restore refused (see Registration safety) holds its name the same way; its 409 carries details.state: "refused" and the restore's details.reason.

Two repositories never share a bound scope. A binding journaled by an earlier build keeps its checkout-directory leaf, which a new registration named like that checkout directory, from the same checkout under the same parent, would derive again; that registration is refused with 409 REPO_SCOPE_IN_USE (details.scope), before anything is opened or provisioned.

The authorization order matters and is part of the contract: the caller is first authorized against the binding the repository has TODAY (Owner on its bound scope), and only then against the parent scope the request named. A caller who may not mutate the existing repository gets 404 REPO_NOT_FOUND — byte-identical to a name that does not exist — so an intra-tenant caller can neither move another scope's repository under one they control nor learn that the name is taken.

Registration safety: what may be registered and what is indexed

Registering a repository makes every file the plane indexes under it readable through /v1/code/*, so path is treated as a capability, not a hint.

Which paths may be registered

The requested path is canonicalized first — .. segments and a symlinked directory resolve to their real target — and every rule below is judged on that canonical path, compared whole component by whole component (/srv/repos2 is not inside /srv/repos).

Always refused, whatever the configuration:

• the filesystem root (/, or a drive root on Windows);
• the system trees /proc, /sys, /dev, /run, /boot, /etc and everything under them, whether or not the path exists;
• the server's data directory and the code plane's own storage (<data_dir>/code), and anything inside them.

A checkout that contains the data directory (a development layout that keeps cortexdb_data/ inside the working tree) is admitted; the data directory's subtree is excluded from every scan and appears in the coverage ledger as Ignored.

CORTEX_CODE_REPO_ROOTS confines registration further:

Roots are resolved on every registration, so a volume mounted after startup starts admitting without a restart.

Refusals answer 422 REPO_PATH_NOT_ALLOWED with details.reason set to relative_path, outside_allowed_roots, no_allowed_roots, filesystem_root, system_path or data_directory. The message never names what a path resolved to. Outside the allowed roots a path that does not exist is refused exactly like one that does, so the answer is not an existence check for the rest of the filesystem; a missing path inside the roots is an ordinary 422 REGISTRATION_FAILED.

The same policy applies when the server restarts: a journaled repository that the current configuration refuses — or whose checkout or stores fail to open — is not restored. It is logged, counted in the startup restore report as a failure, and never stops the other repositories from serving. A registered directory that no longer resolves to itself — a symbolic link now stands in its place, or in place of any directory above it — is not restored either, whatever the link points at; checkouts moved behind a link on purpose must be removed and registered again at their new path. A repository not restored is not served: every read answers 404 REPO_NOT_FOUND and listings omit it. It is still registered, under the binding its journal line records:

• DELETE /v1/code/repos/{name} removes it — journal line, directory and all — for a caller who could have removed it while it was served (Owner on that binding), whether or not anybody governs the tenant root;
• POST /v1/erasures / POST /v1/forget over a scope at or above that binding purge it and count it;
• the name stays taken: registering it again answers 409 REPO_EXISTS (details.state: "refused"), or 404 to a caller who may not mutate it, until it is removed.

Admission is also re-checked every time the checkout is read again. Before each rescan and watcher refresh, and again before a build re-reads the files its scan admitted, the repository root must still resolve to the directory that was admitted and still pass the policy above. A checkout directory replaced by a link (mv app app.old && ln -s other app), or an allowed root re-pointed since registration, fails that check, and the refresh is refused: nothing under the root is read, the repository keeps serving the generation it last published (no file content is served while the refusal stands: context snippets answer "content unavailable"), and the refusal is logged and reported as freshness.root_refused in GET /v1/code/repos and GET /v1/code/repos/{name} until a refresh against the admitted directory publishes. A root that no longer exists is refused the same way. Only this check refuses without fencing reads: a refresh that fails for any other reason — even while the root has also moved — may have failed after the live search indexes began to change, so reads answer 409 CODE_INDEX_REFRESHING until a later refresh publishes.

What is indexed

• A Git worktree (the directory, or an ancestor, holds a Git repository) indexes every file .gitignore does not exclude — including untracked files — minus the built-in vendor and generated skips (node_modules/, target/, dist/, *.min.js, generator banners, …).

• Any other directory indexes only files whose extension has a registered grammar, and skips every hidden file and directory (a leading .). There is no .gitignore there to say what is source. To index more, git init the directory and describe it with a .gitignore.

• Credential files are never read, in either mode. Matched on the path alone, before a byte is opened, case-insensitively:

  • dotenv files: .env, .env.* and *.env — except the templates .env.example, .env.sample and .env.template, which carry names, not values, and are indexed in a Git worktree;
  • private keys and key stores: *.pem, *.key, *.p12, *.pfx, *.jks, *.keystore, *.kdbx, *.ppk, and SSH keys id_rsa, id_dsa, id_ecdsa, id_ed25519 with every variant (id_rsa.pub, id_ed25519_sk, …);
  • credential files: .netrc, _netrc, .pgpass, .npmrc, .pypirc, .git-credentials, .htpasswd, credentials, credentials*.json, client_secret*.json;
  • Terraform state and variables: *.tfstate, *.tfstate.backup, *.tfvars;
  • everything under .ssh/, .gnupg/, .aws/, .azure/, .kube/, .docker/ and .password-store/.

  Each skip is a coverage-ledger row with reason Ignored (counted in coverage.files_skipped of /v1/code/coverage/status), so the exclusion is auditable rather than a silent drop. The list is closed: a secret embedded in a source file (an API key in config.py) is not detected and is indexed like the rest of the file.

• Files over 4 MiB are skipped as TooLarge, and the cap is enforced on the read itself: a file that reports a small size and keeps streaming is never held beyond the cap. The same holds for every later read of the file — parsing, import resolution, search indexing and serving context. Files with a NUL byte in their first 8 KiB are skipped as Binary.

• Symbolic links in the checkout are not followed. The walk never enters a linked directory, and the filesystem watcher reports no event from behind an in-repository directory link. A file is read only if it really is at its path under the root: it is opened without following a link at its path, so one swapped for a link (or a FIFO or device) after the walk saw it is skipped, and the open is refused when it resolved through a link anywhere above the file — a directory swapped for a link after the walk listed it (mv src src.old && ln -s ../team-b/src src). On Linux and Windows the open handle is asked where its file is, so nothing races that check; elsewhere the path is resolved again after the open, which misses only a link swapped out again in between. A directory swapped while the walk was listing it can put names from the other tree in the coverage ledger, never their content.

• On Windows, a reparse point that is not a link — deduplication, HSM and cloud-file placeholders, container layers — is read through its filter, as the file's real content, never as the stub underneath; one whose filter is not installed is skipped.

• A build re-reads a file only as the bytes its scan hashed. A file rewritten or swapped between the scan and the re-read is left out of that build rather than parsed or indexed as something else; the next refresh picks up the change.

Scan limits

Every scan — registration, restart, a rescan and every watcher-triggered refresh — is bounded by compiled-in limits, far above any real repository (the largest measured, 925,925 files, scanned in 16 s):

Crossing one fails the scan; it never indexes part of a tree. A registration answers 422 REGISTRATION_FAILED with details.limit and details.max naming the bound, and nothing is journaled — register a narrower directory. A watcher refresh that crosses one records the error in the repository's freshness.last_error and reads answer 409 CODE_INDEX_REFRESHING until a later refresh succeeds or the repository is removed.

Scans also never descend into a kernel pseudo-filesystem mounted inside the tree — a container root filesystem with a live proc/, or a bind mount of the host's /sys or /dev — whose files report meaningless sizes and can stream without end. On Linux the scan finds these in the mount table by filesystem type (proc, sysfs, devtmpfs and the kernel filesystems mounted under them), whatever path they are mounted at, and records each mount point as an Ignored directory in the coverage ledger. A repository whose root is itself inside one fails to scan. The filesystem watcher applies the same guard: when such a mount is inside the tree it watches every other directory one by one, adding new ones as they appear, and never walks or watches the mount, so registering a live container root filesystem does not fail on a process directory under its proc/ that exits, or cannot be read, while the watcher starts. Other platforms have no mount table to consult; there, as everywhere, only regular files are ever read.

Removal and erasure

DELETE /v1/code/repos/{name} frees the bytes — of a served repository, and of a journaled one the last restore refused. It removes the map entry, writes the durable removal intent, quiesces the watcher and the refresh lock, waits for the stores to be released, deletes <data_dir>/code/<blake3(tenant)[..16]>/<name>/, and rewrites the registry journal with no line for that repository — so the journal no longer names the checkout path either (RTBF). The response carries the report:

{ "removed": "myrepo",
  "purge": { "tenant": "…", "name": "myrepo", "scope": "org:acme/team:eng/repo:myrepo-9e49cb54ea4b",
             "directory": "…/code/1a2b3c4d5e6f7890/myrepo",
             "removed": true, "bytes_freed": 41283904, "registry_compacted": true } }

Two honest failure modes, neither of which loses the request:

• 503 REPO_PURGE_BUSY — de-registered, but a reader still holds the stores (details.retry_after_ms says how long the purge waited). The removal intent is already durable: retrying finishes it, and so does the next boot. The retry is authorized against the scope the repository was bound to, so whoever could remove it can finish removing it.
• 500 REPO_PURGE_FAILED — de-registered, directory survives (details.path). Same recovery.

At boot the plane replays the journal and then reconciles orphan directories: any <data_dir>/code/<hash16>/<name> that no registered repository claims is removed, and the counts appear in the startup restore report. A crashed purge cannot leave data behind forever.

Forget and erasure reach the code plane. The plane registers itself with the forget engine as a scope-erasure participant, wired at the one router composition site, so:

• POST /v1/erasures over a scope purges every repository bound under that scope and reports summary.deleted_code_repos; the preview reports code_repos_to_delete beforehand.
• POST /v1/forget removes them only under raw-event parity — repositories are sources, not derived layers — i.e. confirm_all: true, an empty selector, cascade: "redact_events", and layers either empty or containing "code". That request reports deleted.code_repos. Any other shape keeps them and reports deleted.code_repos_retained plus a warning naming the request shape that would remove them.
• A participant failure is 502 FORGET_BACKEND_FAILED / ERASURE_BACKEND_FAILED. Nothing is reported deleted that was not; the purge is idempotent, so the identical request can be retried.
• A failed purge keeps failing until the bytes are gone. A repository the purge de-registered but could not free is recorded as a PENDING purge and stays listed under its scope, so the retry finds it, retries the directory removal, and reports the failure again rather than reporting a clean scope over a checkout that is still on disk. GET /v1/admin/metrics exposes the count as code.pending_purges; it converges to zero, at the latest on the next boot sweep.
• The erasure participant runs on a blocking worker (spawn_blocking), not on the request thread: it waits for readers and then deletes whole trees.

Upgrade notes: registry v2

The repository registry journal (<data_dir>/code/repositories.v1.jsonl, filename unchanged) is now version 2: each line carries the parent scope the repository is bound under. Replay accepts both versions — a v1 line has no scope and binds under the tenant root org:<billing id>, which is exactly what it meant when it was written — and every new line, including the compaction that a removal performs, is written as v2.

Rolling back to a binary older than this release is therefore not transparent: it rejects a v2 line loudly (line N uses unsupported version 2) rather than misreading it. The recovery is to delete the journal: the plane is default-off, <data_dir>/code is removable per the ADR rollback (nothing else references it, and every repository is re-registerable from its checkout), and a deleted journal replays as "no repositories".

Upgrade notes: governed scopes and name leaves

• Repositories registered under a scope nobody registered go dark for verified tokens. Before this release a registration created no roster, so a repository bound under an unregistered scope was served to every token in its tenant. It now answers 404 REPO_NOT_FOUND to verified per-actor callers until its parent is governed. The remedy is to register the parent scope (POST /v1/scopes) and seat its members (PUT /v1/scopes/members); the operator key still sees and can remove it.
• New registrations bind repo:<name>-<12 hex>. Journal lines written by earlier builds carry no leaf field and keep the checkout-directory leaf (repo:<checkout directory>-<12 hex>) until the repository is removed and registered again, so no authorization boundary moves at boot. When two such repositories share one bound scope, boot logs a warning naming them, and a new registration whose name leaf would equal such a binding is refused with 409 REPO_SCOPE_IN_USE. New lines add "leaf": "name" and stay version 2: an older binary ignores the field and opens the repository under its old leaf.

Upgrade notes: registration safety

• Some journaled repositories may not come back. Restore applies the registration path policy (Registration safety). A repository registered from a path it now refuses — inside the data directory (for example /data/<checkout> in the Docker image), or outside a newly set CORTEX_CODE_REPO_ROOTS — is not restored: the startup log carries a warning and the restore report a failure naming it. It is not served, but stays registered: its Owner may DELETE it, scope erasure purges it, and its name stays taken until then. To keep serving it, move the checkout inside the policy (e.g. mount it at /repos), remove the old registration and register it again.
• Directories that are not Git worktrees index less. On the rebuild below, such a repository indexes only files with a registered grammar and skips hidden files and directories, so stats.files may drop; git init the directory and describe it with a .gitignore to index more.
• Credential files drop out of every repository on that rebuild, tracked or not (see What is indexed).

Upgrade notes: extractor generation 4

The syntax extractor is now generation 4 (C/C++ declarators, qualified members, export-macro classes, gtest bodies, qualified callees, the .h sniff). A repository's persisted runtime state records the generation it was built with, and a restore that finds another one rebuilds the repository once instead of serving the old graph — on the first boot after the upgrade every registered repository the current path policy admits is fully re-indexed, synchronously at boot before the server starts serving (roughly one registration's time per repository); one it refuses is not restored at all (see Upgrade notes: registration safety). The rebuild publishes into a fresh generation, so nothing of the old graph survives into the new one. Portable bundles exported by an older server are refused on import (GenerationMismatch); export them again.

The SCIP importer is now generation 2 (call edges, typed ranges). Blocks imported before the upgrade stay attached and keep certifying, but carry no call edges: re-run POST /v1/code/import/scip to get compiler-tier edges in graph/query and impact. The re-import leaves the old block attached next to the new one until the file changes; certificates count each occurrence once, and replacing a file's older blocks on import is a follow-up.

Limits worth knowing

• Files nested deeper than 200 syntax levels are skipped with a reason (machine-generated and adversarial inputs; real code is orders of magnitude shallower). Skips are coverage-ledger data.
• Per-file parse and extraction budgets, fact caps and query row budgets are all hard; crossing one is reported, never silently truncated.
• Every scan is bounded by entries, depth, bytes read and wall clock (see Scan limits); crossing a bound fails the scan.
• History memory walks first-parent history from HEAD over the last 128 commits (a fixed window, not configurable). A commit touching more than 64 files is skipped as bulk but still uses a step of the window (history_bulk_skipped), and each path keeps at most 8 commit summaries. A shallow clone is walked to its boundary: a depth-N clone yields at most N-1 commits, and stats report history_shallow: true. history_commits_scanned counts commits that changed files, history_window is the window, and history_window_exhausted: true means older history exists beyond it.
• Cluster-mode serving and auto-SCIP are deliberately out of the launch scope. Code vectors are opt-in per deployment (CORTEX_CODE_EMBEDDING_PROVIDER), not a launch default.