Skip to content

Architecture

basemind is a single Rust crate with one binary (basemind) serving three roles: basemind scan indexes a workspace, basemind serve runs the MCP stdio server, and the daemon (basemind comms daemon, auto-spawned; singleton per user) is the sole writer managing the global cache.

Your project (code + documents + git history)
↓
basemind scan (or incremental watch)
├─ Route each file to a tier: code -> code map; prose, data and config -> document tier
├─ Tree-sitter parser pool (300+ languages)
├─ Extract L1 (outlines: symbols, signatures, imports)
├─ Extract L2 (calls: callee, byte offset, line/col) [if eager_l2=true]
├─ Extract L3 (structural hash of symbol bodies)
├─ Per-file trigram bloom filter (grep prefilter), built from bytes already read
└─ Forward writes to basemind daemon (IPC over UDS)
├─ Daemon writes content-addressed msgpack blobs
└─ Daemon writes Fjall LSM index (sole Fjall writer per machine)
↓
basemind serve (stdio relay to the daemon)
├─ Daemon hosts the MCP server and the read stack for every session
├─ Index scans (references, callers, implementations) run in the daemon
├─ N sessions per repo share one index; none opens Fjall itself
├─ Answer MCP tool requests from caches, resident name indexes and the on-disk index
└─ Daemon watches the filesystem and rescans what changed

basemind persists project state in a machine-global cache under ~/.local/share/basemind/ (or $BASEMIND_DATA_HOME), keyed by workspace root. All repos/worktrees on one machine share a single content-addressed blob store and a daemon-managed Fjall writer.

Location: ~/.local/share/basemind/cache/blobs/<hash>.{l1,l2,l3}.msgpack

Each extraction tier is a separate msgpack blob, keyed by content hash. This enables deduplication across files with identical bodies.

  • L1 (always): outlines with symbols, signatures, imports, doc comments
  • L2 (if eager_l2=true): call sites (callee name, byte offset, line/column)
  • L3: structural hash of each symbol’s body (used by git_log with symbol)

Document and chunk artifacts share the same store under other suffixes (.fm, .chunk, .rref, .doc). Every file entry and L1 blob records the extractor epoch it was produced under, so a fix that changes what an unchanged file should contain re-extracts only the stale files. Blobs orphaned by a migration are reclaimed by a tier-aware, reference-counted sweep (see Upgrading).

Blobs are immutable and content-addressed, so they can be safely shared across multiple views (scans) and repos.

Location: ~/.local/share/basemind/cache/workspaces/<workspace_key>/views/<view>/index.fjall/

A disk-backed, append-only LSM tree (Fjall) provides fast lookups. Fjall admits one holder per directory, so the daemon is the only process that opens it: it writes, and it answers the index reads that sessions forward to it (code_refs call-site and implementation scans, and the other code tools through the read stack it hosts):

Keyspace Purpose
meta Schema version and constants
symbols_by_path Per-file symbol lookup
symbols_by_name Reserved name-prefix partition, written on every upsert; code_symbols is answered by the resident term index
calls_by_path Per-file call lookup
calls_by_callee Callee-keyed call sites; a resident dictionary of distinct names routes substring queries to the matching key ranges (drives code_refs)
imports_by_path Per-file import lookup
imports_by_module Reserved module-prefix partition; dependents is answered by the resident term index
implementations_by_path Per-file implementation lookup
implementations_by_trait Trait-keyed implementations, with its own name dictionary (drives implementations)
refs_by_def / refs_by_path Scope- and import-resolved reference edges, keyed by defining site and by use file
code_bm25_postings / code_bm25_by_path Term postings and their per-file companion for the code-search keyword lane
grep_bloom Per-file trigram bloom filter and the size and mtime it was built under; prefilter for code_grep
embeddings Reserved for future in-Fjall vector index
memory_by_key Agent memory namespaced by (scope, visibility, owner)

All composite keys are length-prefixed (u16:len ‖ bytes), ensuring a prefix like Foo never spills into Foobar.

Schema version is stamped in meta. On version mismatch, the next basemind scan auto-wipes and rebuilds.

Location: ~/.local/share/basemind/cache/workspaces/<workspace_key>/lance/

Powers semantic search (code search, document search, memory search). Vectors are indexed for fast KNN queries; on-disk format is LanceDB’s columnar structure. Writes go through the daemon.

basemind scan and basemind watch (incremental) share the same per-file pipeline:

Walker (gitignore-aware) → filter by glob + size cap
↓
rayon par_iter (pool size: [resources] scan_threads, default one per core)
↓
process_file(rel_path, file_contents):
1. lang::detect() ← tree-sitter language pack, [languages.<name>] overrides first;
no grammar -> document tier
2. Extract L1 (outline) ← tree-sitter queries
3. Extract L2 (calls) ← if eager_l2=true
4. Write L1 to blob store
5. Write L2 to blob store ← if eager_l2=true
5b. Build grep bloom filter ← from the bytes already read
6. Index lookup:
a. Read existing entries (if re-scan)
b. Stage all deletes + inserts in one atomic Fjall batch
7. Per-file commit ← Fjall handles cross-thread locking
↓
collect FileResult { rel, l1_hash, l2_hash?, … }
↓
apply_outcomes:
· Write Index meta
· Remove stale files via IndexWriter::remove_file

Per-file commit — Every file commits its Fjall batch before returning. Fjall handles cross-thread locking; the scanner does not.

Atomic upsert — IndexWriter::upsert_file is read-before-write: it reads existing entries first to derive secondary-index keys for deletion, then stages all deletes + inserts in one batch. This prevents torn state on re-scan.

Eager L2 cost — Eager L2 (call-site extraction) is on by default and adds to scan time. Set eager_l2 = false to skip it for a faster scan, trading away reference search.

Bounded memory — Document extraction and code-chunk embedding pass a footprint gate ([resources] max_footprint_mb), stack-graph resolution is capped per file and to two concurrent builds, and ONNX defaults to the CPU provider. See Memory and resources.

scan_paths removal mirror — When a file disappears between scans, scan_paths calls IndexWriter::remove_file so secondary indexes don’t leak stale entries.

Per-file symbol table with signatures, line/column locations, import statements, and doc comments. Generated by tree-sitter queries (hand-written overrides at src/queries/<language>.scm, falling back to tree-sitter-language-pack’s tags.scm).

Used by: code_outline, code_symbols and code_refs with relation: "definition".

Every call site in a file: callee name, byte offset, line/column. Extracted only if eager_l2 = true.

Used by: code_refs.

Disabling it speeds up the scan but makes reference search unavailable.

Cryptographic hash of each symbol’s body. Used to detect when a symbol’s implementation changed across commits.

Used by: git_log with path and symbol.

When [documents] enabled = true, scanner_docs.rs runs a separate pass. The code map holds code only: Markdown, reStructuredText, AsciiDoc, vimdoc, CSV, JSON, YAML, TOML, INI, XML, .properties, .env, diffs, .gitignore / .gitattributes and Fluent files are routed here instead of being outlined as symbols, gated by [documents] include / exclude and the MIME allowlist. A path that changes tier is purged from the old one on the next scan.

  • Reads PDFs, Office files, HTML, images, email, and the prose, data and config files above
  • Extracts text (OCR for images and scanned PDFs needs the documents-ocr Cargo feature, which the published binaries include)
  • Chunks and embeds via LanceDB
  • Searched with docs_search

basemind uses the tree-sitter language pack (TSLP) to detect and parse 300+ languages. Language detection is dynamic:

[languages.<name>] overrides (file names, then extensions; enabled = false opts a grammar out)
↓
File extension → tree_sitter_language_pack::detect_language
↓
LangId = &'static str (the pack name, e.g. "rust", "python", "typescript")
↓
Parser from pool (lazy init, reused across files)
↓
Query (hand-written override OR TSLP tags.scm fallback)
↓
Extract symbols + calls

A file whose grammar is disabled or unrecognised falls through to the document tier. [languages.<name>] in basemind.toml maps extra extensions and file names onto a grammar, disables a misdetected one (.txt as vimdoc, .conf as nginx), and can preload a grammar for basemind lang install.

All languages with a tags.scm query in TSLP (~100) work out of the box. For richer extraction, add a .scm override at src/queries/<pack-name>.scm.

basemind git commands are powered by gix (a pure-Rust Git implementation) with a git-history index for microsecond latency:

Location: ~/.local/share/basemind/cache/workspaces/<workspace_key>/git-cache/

Two-tier caching:

  • In-process LRU (1024 entries per category)
  • Disk store (SHA-keyed):
    • commit_files/<sha>.msgpack — file list for a commit (immutable, never stales)
    • log/<head_sha>__<scope>.msgpack — commit log (keyed by HEAD, rolls off naturally)
    • blame/<sha>__<path_hash>.msgpack — blame data (immutable, never stales)

History queries (git_log, git_blame) are posting-list lookups over this index, answering in tens of microseconds flat.

The index is a pure accelerator: it’s used only when fresh (last_indexed_head == HEAD) and otherwise walks history directly, so it can never serve stale results. It rebuilds automatically on history rewrite (filter-repo, rebase, force-push).

Two persistent schemas, both tracking RELEASE_MINOR from src/version.rs:

  • INDEX_SCHEMA_VER in src/index/mod.rs — Fjall keyspace format
  • SCHEMA_VER in src/extract/mod.rs — msgpack blob format

Patch release (0.1.0 → 0.1.1): RELEASE_MINOR unchanged. Blobs + index must stay compatible — no format changes. A fix that changes what extraction should produce for an unchanged file bumps extract::EXTRACT_EPOCH instead; only the files whose entry or blob predates it are re-extracted. A new derived index (such as grep_bloom) is stamped with its own version and rebuilt lazily on the next scan.

Minor release (0.1.x → 0.2.0): RELEASE_MINOR increments. On next basemind scan, the stale cache is auto-wiped and rebuilt from source. Users are notified in release notes.

Wipe-on-mismatch is the explicit migration story for minor releases. No manual steps — basemind scan handles it, then sweeps the blobs it orphaned.

Global cache at ~/.local/share/basemind/ (or $BASEMIND_DATA_HOME):

~/.local/share/basemind/cache/
├── blobs/
│ ├── abc123….l1.msgpack # Outline: symbols, signatures, imports
│ ├── abc123….l2.msgpack # Calls (if eager_l2=true)
│ ├── abc123….l3.msgpack # Structural hashes
│ └── (shared across all repos/worktrees via content hash)
└── workspaces/
└── <workspace_key>/ # blake3 of the canonical worktree root
├── views/<view>/
│ ├── index.msgpack
│ └── index.fjall/ # Fjall LSM index (one writer per workspace)
├── git-history.fjall/ # Precomputed git-history index
├── git-cache/ # Cached git payloads
├── lance/ # Vector store (documents, memory, code search)
├── workspace.json # Worktree root this key was derived from
├── status.json # File counts + scan age for the statusline
└── .lock, .lock.meta # Writer lock and its holder
Per-repo configuration (repo-local):
repo/basemind.toml # Configuration

The daemon manages the global cache; individual repos hold no .basemind/ directory.

basemind serve exposes a stdio MCP server with one tool per job (ADR-0018): code_outline, code_symbols, code_read, code_grep, code_search, code_refs, find_files, docs_search, git_log, git_blame, git_diff, memory_save, memory_search, index_status and index_rescan. None takes a mode. comms builds add agents, which still dispatches on a required mode. Every tool definition carries _meta["anthropic/alwaysLoad"]: true, so the tools are always loaded; tools/list for the fifteen tools is about 17 KB (about 4.1k tokens), held under 20,000 bytes by a test. All paths are byte-precise repo-relative (RelPath).

Tool conventions:

  • Paths: always RelPath (repo-relative), never arbitrary strings
  • Arguments: each tool has its own exact schema (src/mcp/types_tools.rs); an unknown argument is rejected with a did-you-mean suggestion (src/mcp/strict.rs)
  • Positions: line, end_line and column are 1-based everywhere
  • Lists: take limit and cursor, return next_cursor, and are trimmed to max_tokens (default 6000), setting budgeted
  • Errors: a failure the caller can fix is an isError tool result naming the fix (src/mcp/tool_contract.rs)
  • Results: inline; a server-side Task only for a call still running after 3 s
  • Index scans: internally use scan_cap = limit * 8 (at least 2000) to bound work on common names
  • Descriptions: say when to use the tool and carry one example call
  • Stability: new fields are additive with #[serde(default)]
  • Annotations: every tool has a title and read-only / destructive / idempotent / open-world hints, exact per tool: only memory_save and index_rescan write, and none is destructive
  • Telemetry: keyed by tool name
  • Resources: basemind://status, basemind://outline/{path} and basemind://memory/{key} re-expose tool output on demand (see MCP tools)

Tool bodies live in src/mcp/helpers*.rs (sliced by area: helpers_documents.rs, helpers_calls.rs, helpers_grep.rs, etc.); src/mcp/tools*.rs contain only thin #[tool] shims.

The background daemon (singleton per user) manages agent communication and shared state:

  • Threads: per-repo, per-workspace, or global scope; agents join explicitly via the agents tool’s join mode
  • Messages: front-matter stored in messages_by_thread, bodies in message_body (fetched on demand for efficiency)
  • Registry: tracks the workspaces, worktrees and branches the daemon has seen; listed by basemind workspace
  • Memory: namespaced by (scope, visibility, owner) — group memory is shared, individual memory is private to one agent
  • Vectors: LanceDB stores embeddings for memory + document search

The broker protocol (version 5) correlates every request after the handshake with a reply id, bounds every wait (10 s for requests, 30 s for the handshake, 180 s in the stdio relay), runs all store access off the async workers (spawn_blocking, with a write gate for mutations), and deduplicates keyed posts for an hour. On Unix the daemon accepts connections before its store opens and answers only Ping until it does. After a backend restart the relay re-sends reads and keyed posts once and fails everything else with a retryable error. The detached daemon logs to daemon.log (rotated at 8 MiB). See Agent coordination and ADR-0013.

Why a daemon? Content-addressed blobs and a single Fjall writer are the key wins: all repos/worktrees on one machine share blob deduplication, and N sessions can read concurrently from a single index without downgrade-to-read-only issues. The daemon is the sole Fjall writer and enforces this via socket-bind-as-lock (singleton enforcement).

The daemon’s socket, lock, log and comms store live in a protocol-generation subdirectory of the comms directory (comms/g1/). A basemind binary only looks inside its own protocol generation, so two installed releases of different generations each run their own daemon instead of replacing each other’s. Releases up to 0.33 use the comms directory itself. The daemon registry is scoped the same way, so comms stop --all and comms doctor cover their own generation’s daemons only. The workspace index, memory and git-history stores are not per generation: a second daemon cannot serve a workspace the first one holds. A 0.33 plugin launcher still kills other versions’ processes started from the plugin cache, so the two coexist fully only once it is gone. With BASEMIND_ALLOW_HTTP, the second daemon to start binds an OS-assigned port instead of the default; read it from http.addr. See Upgrading.