Architecture
basemind is a single Rust crate with one binary (basemind) serving three roles: basemind scan indexes a workspace, basemind serve runs the MCP stdio server, and the daemon (basemind comms daemon, auto-spawned; singleton per user) is the sole writer managing the global cache.
High-Level Pipeline
Section titled “High-Level Pipeline”Your project (code + documents + git history) ↓basemind scan (or incremental watch) ├─ Route each file to a tier: code -> code map; prose, data and config -> document tier ├─ Tree-sitter parser pool (300+ languages) ├─ Extract L1 (outlines: symbols, signatures, imports) ├─ Extract L2 (calls: callee, byte offset, line/col) [if eager_l2=true] ├─ Extract L3 (structural hash of symbol bodies) ├─ Per-file trigram bloom filter (grep prefilter), built from bytes already read └─ Forward writes to basemind daemon (IPC over UDS) ├─ Daemon writes content-addressed msgpack blobs └─ Daemon writes Fjall LSM index (sole Fjall writer per machine) ↓basemind serve (stdio relay to the daemon) ├─ Daemon hosts the MCP server and the read stack for every session ├─ Index scans (references, callers, implementations) run in the daemon ├─ N sessions per repo share one index; none opens Fjall itself ├─ Answer MCP tool requests from caches, resident name indexes and the on-disk index └─ Daemon watches the filesystem and rescans what changedStorage: Global Cache
Section titled “Storage: Global Cache”basemind persists project state in a machine-global cache under ~/.local/share/basemind/ (or $BASEMIND_DATA_HOME), keyed by workspace root. All repos/worktrees on one machine share a single content-addressed blob store and a daemon-managed Fjall writer.
Content-Addressed Msgpack Blobs
Section titled “Content-Addressed Msgpack Blobs”Location: ~/.local/share/basemind/cache/blobs/<hash>.{l1,l2,l3}.msgpack
Each extraction tier is a separate msgpack blob, keyed by content hash. This enables deduplication across files with identical bodies.
- L1 (always): outlines with symbols, signatures, imports, doc comments
- L2 (if eager_l2=true): call sites (callee name, byte offset, line/column)
- L3: structural hash of each symbol’s body (used by
git_logwithsymbol)
Document and chunk artifacts share the same store under other suffixes (.fm, .chunk, .rref,
.doc). Every file entry and L1 blob records the extractor epoch it was produced under, so a fix
that changes what an unchanged file should contain re-extracts only the stale files. Blobs
orphaned by a migration are reclaimed by a tier-aware, reference-counted sweep (see
Upgrading).
Blobs are immutable and content-addressed, so they can be safely shared across multiple views (scans) and repos.
Fjall LSM Inverted Index
Section titled “Fjall LSM Inverted Index”Location: ~/.local/share/basemind/cache/workspaces/<workspace_key>/views/<view>/index.fjall/
A disk-backed, append-only LSM tree (Fjall) provides fast lookups. Fjall admits one holder per
directory, so the daemon is the only process that opens it: it writes, and it answers the index
reads that sessions forward to it (code_refs call-site and implementation scans, and the other
code tools through the read stack it hosts):
| Keyspace | Purpose |
|---|---|
meta |
Schema version and constants |
symbols_by_path |
Per-file symbol lookup |
symbols_by_name |
Reserved name-prefix partition, written on every upsert; code_symbols is answered by the resident term index |
calls_by_path |
Per-file call lookup |
calls_by_callee |
Callee-keyed call sites; a resident dictionary of distinct names routes substring queries to the matching key ranges (drives code_refs) |
imports_by_path |
Per-file import lookup |
imports_by_module |
Reserved module-prefix partition; dependents is answered by the resident term index |
implementations_by_path |
Per-file implementation lookup |
implementations_by_trait |
Trait-keyed implementations, with its own name dictionary (drives implementations) |
refs_by_def / refs_by_path |
Scope- and import-resolved reference edges, keyed by defining site and by use file |
code_bm25_postings / code_bm25_by_path |
Term postings and their per-file companion for the code-search keyword lane |
grep_bloom |
Per-file trigram bloom filter and the size and mtime it was built under; prefilter for code_grep |
embeddings |
Reserved for future in-Fjall vector index |
memory_by_key |
Agent memory namespaced by (scope, visibility, owner) |
All composite keys are length-prefixed (u16:len ‖ bytes), ensuring a prefix like Foo never spills into Foobar.
Schema version is stamped in meta. On version mismatch, the next basemind scan auto-wipes and rebuilds.
LanceDB Vector Store
Section titled “LanceDB Vector Store”Location: ~/.local/share/basemind/cache/workspaces/<workspace_key>/lance/
Powers semantic search (code search, document search, memory search). Vectors are indexed for fast KNN queries; on-disk format is LanceDB’s columnar structure. Writes go through the daemon.
Scan Pipeline
Section titled “Scan Pipeline”basemind scan and basemind watch (incremental) share the same per-file pipeline:
Walker (gitignore-aware) → filter by glob + size cap ↓rayon par_iter (pool size: [resources] scan_threads, default one per core) ↓process_file(rel_path, file_contents): 1. lang::detect() ← tree-sitter language pack, [languages.<name>] overrides first; no grammar -> document tier 2. Extract L1 (outline) ← tree-sitter queries 3. Extract L2 (calls) ← if eager_l2=true 4. Write L1 to blob store 5. Write L2 to blob store ← if eager_l2=true 5b. Build grep bloom filter ← from the bytes already read 6. Index lookup: a. Read existing entries (if re-scan) b. Stage all deletes + inserts in one atomic Fjall batch 7. Per-file commit ← Fjall handles cross-thread locking ↓collect FileResult { rel, l1_hash, l2_hash?, … } ↓apply_outcomes: · Write Index meta · Remove stale files via IndexWriter::remove_fileKey Invariants
Section titled “Key Invariants”Per-file commit — Every file commits its Fjall batch before returning. Fjall handles cross-thread locking; the scanner does not.
Atomic upsert — IndexWriter::upsert_file is read-before-write: it reads existing entries first to derive secondary-index keys for deletion, then stages all deletes + inserts in one batch. This prevents torn state on re-scan.
Eager L2 cost — Eager L2 (call-site extraction) is on by default and adds to scan time. Set eager_l2 = false to skip it for a faster scan, trading away reference search.
Bounded memory — Document extraction and code-chunk embedding pass a footprint gate ([resources] max_footprint_mb), stack-graph resolution is capped per file and to two concurrent builds, and ONNX defaults to the CPU provider. See Memory and resources.
scan_paths removal mirror — When a file disappears between scans, scan_paths calls IndexWriter::remove_file so secondary indexes don’t leak stale entries.
Extraction Tiers
Section titled “Extraction Tiers”L1: Outlines (Always)
Section titled “L1: Outlines (Always)”Per-file symbol table with signatures, line/column locations, import statements, and doc comments. Generated by tree-sitter queries (hand-written overrides at src/queries/<language>.scm, falling back to tree-sitter-language-pack’s tags.scm).
Used by: code_outline, code_symbols and code_refs with relation: "definition".
L2: Call Sites (Optional, Default On)
Section titled “L2: Call Sites (Optional, Default On)”Every call site in a file: callee name, byte offset, line/column. Extracted only if eager_l2 = true.
Used by: code_refs.
Disabling it speeds up the scan but makes reference search unavailable.
L3: Structural Hashes
Section titled “L3: Structural Hashes”Cryptographic hash of each symbol’s body. Used to detect when a symbol’s implementation changed across commits.
Used by: git_log with path and symbol.
Document Extraction (xberg)
Section titled “Document Extraction (xberg)”When [documents] enabled = true, scanner_docs.rs runs a separate pass. The code map holds code
only: Markdown, reStructuredText, AsciiDoc, vimdoc, CSV, JSON, YAML, TOML, INI, XML, .properties,
.env, diffs, .gitignore / .gitattributes and Fluent files are routed here instead of being
outlined as symbols, gated by [documents] include / exclude and the MIME allowlist. A path that
changes tier is purged from the old one on the next scan.
- Reads PDFs, Office files, HTML, images, email, and the prose, data and config files above
- Extracts text (OCR for images and scanned PDFs needs the
documents-ocrCargo feature, which the published binaries include) - Chunks and embeds via LanceDB
- Searched with
docs_search
Language Support
Section titled “Language Support”basemind uses the tree-sitter language pack (TSLP) to detect and parse 300+ languages. Language detection is dynamic:
[languages.<name>] overrides (file names, then extensions; enabled = false opts a grammar out) ↓File extension → tree_sitter_language_pack::detect_language ↓LangId = &'static str (the pack name, e.g. "rust", "python", "typescript") ↓Parser from pool (lazy init, reused across files) ↓Query (hand-written override OR TSLP tags.scm fallback) ↓Extract symbols + callsA file whose grammar is disabled or unrecognised falls through to the document tier. [languages.<name>]
in basemind.toml maps extra extensions and file names onto a grammar, disables a misdetected one
(.txt as vimdoc, .conf as nginx), and can preload a grammar for basemind lang install.
All languages with a tags.scm query in TSLP (~100) work out of the box. For richer extraction, add a .scm override at src/queries/<pack-name>.scm.
Git Layer
Section titled “Git Layer”basemind git commands are powered by gix (a pure-Rust Git implementation) with a git-history index for microsecond latency:
Git-History Index
Section titled “Git-History Index”Location: ~/.local/share/basemind/cache/workspaces/<workspace_key>/git-cache/
Two-tier caching:
- In-process LRU (1024 entries per category)
- Disk store (SHA-keyed):
commit_files/<sha>.msgpack— file list for a commit (immutable, never stales)log/<head_sha>__<scope>.msgpack— commit log (keyed by HEAD, rolls off naturally)blame/<sha>__<path_hash>.msgpack— blame data (immutable, never stales)
History queries (git_log, git_blame) are posting-list lookups over this index, answering in tens of microseconds flat.
The index is a pure accelerator: it’s used only when fresh (last_indexed_head == HEAD) and otherwise walks history directly, so it can never serve stale results. It rebuilds automatically on history rewrite (filter-repo, rebase, force-push).
Schema Versioning
Section titled “Schema Versioning”Two persistent schemas, both tracking RELEASE_MINOR from src/version.rs:
INDEX_SCHEMA_VERinsrc/index/mod.rs— Fjall keyspace formatSCHEMA_VERinsrc/extract/mod.rs— msgpack blob format
Versioning Rules
Section titled “Versioning Rules”Patch release (0.1.0 → 0.1.1): RELEASE_MINOR unchanged. Blobs + index must stay compatible — no format changes. A fix that changes what extraction should produce for an unchanged file bumps extract::EXTRACT_EPOCH instead; only the files whose entry or blob predates it are re-extracted. A new derived index (such as grep_bloom) is stamped with its own version and rebuilt lazily on the next scan.
Minor release (0.1.x → 0.2.0): RELEASE_MINOR increments. On next basemind scan, the stale cache is auto-wiped and rebuilt from source. Users are notified in release notes.
Wipe-on-mismatch is the explicit migration story for minor releases. No manual steps — basemind scan handles it, then sweeps the blobs it orphaned.
Directory Layout
Section titled “Directory Layout”Global cache at ~/.local/share/basemind/ (or $BASEMIND_DATA_HOME):
~/.local/share/basemind/cache/├── blobs/│ ├── abc123….l1.msgpack # Outline: symbols, signatures, imports│ ├── abc123….l2.msgpack # Calls (if eager_l2=true)│ ├── abc123….l3.msgpack # Structural hashes│ └── (shared across all repos/worktrees via content hash)└── workspaces/ └── <workspace_key>/ # blake3 of the canonical worktree root ├── views/<view>/ │ ├── index.msgpack │ └── index.fjall/ # Fjall LSM index (one writer per workspace) ├── git-history.fjall/ # Precomputed git-history index ├── git-cache/ # Cached git payloads ├── lance/ # Vector store (documents, memory, code search) ├── workspace.json # Worktree root this key was derived from ├── status.json # File counts + scan age for the statusline └── .lock, .lock.meta # Writer lock and its holder
Per-repo configuration (repo-local):repo/basemind.toml # ConfigurationThe daemon manages the global cache; individual repos hold no .basemind/ directory.
MCP Surface
Section titled “MCP Surface”basemind serve exposes a stdio MCP server with one tool per job (ADR-0018): code_outline,
code_symbols, code_read, code_grep, code_search, code_refs, find_files, docs_search,
git_log, git_blame, git_diff, memory_save, memory_search, index_status and
index_rescan. None takes a mode. comms builds add agents, which still
dispatches on a required mode. Every tool definition carries _meta["anthropic/alwaysLoad"]: true,
so the tools are always loaded; tools/list for the fifteen tools is about 17 KB (about 4.1k
tokens), held under 20,000 bytes by a test. All paths are byte-precise repo-relative (RelPath).
Tool conventions:
- Paths: always
RelPath(repo-relative), never arbitrary strings - Arguments: each tool has its own exact schema (
src/mcp/types_tools.rs); an unknown argument is rejected with a did-you-mean suggestion (src/mcp/strict.rs) - Positions:
line,end_lineandcolumnare 1-based everywhere - Lists: take
limitandcursor, returnnext_cursor, and are trimmed tomax_tokens(default 6000), settingbudgeted - Errors: a failure the caller can fix is an
isErrortool result naming the fix (src/mcp/tool_contract.rs) - Results: inline; a server-side Task only for a call still running after 3 s
- Index scans: internally use
scan_cap = limit * 8(at least 2000) to bound work on common names - Descriptions: say when to use the tool and carry one example call
- Stability: new fields are additive with
#[serde(default)] - Annotations: every tool has a title and read-only / destructive / idempotent / open-world hints, exact per tool: only
memory_saveandindex_rescanwrite, and none is destructive - Telemetry: keyed by tool name
- Resources:
basemind://status,basemind://outline/{path}andbasemind://memory/{key}re-expose tool output on demand (see MCP tools)
Tool bodies live in src/mcp/helpers*.rs (sliced by area: helpers_documents.rs, helpers_calls.rs, helpers_grep.rs, etc.); src/mcp/tools*.rs contain only thin #[tool] shims.
Agent Comms & Shared Memory
Section titled “Agent Comms & Shared Memory”The background daemon (singleton per user) manages agent communication and shared state:
- Threads: per-repo, per-workspace, or global scope; agents join explicitly via the
agentstool’sjoinmode - Messages: front-matter stored in
messages_by_thread, bodies inmessage_body(fetched on demand for efficiency) - Registry: tracks the workspaces, worktrees and branches the daemon has seen; listed by
basemind workspace - Memory: namespaced by
(scope, visibility, owner)— group memory is shared, individual memory is private to one agent - Vectors: LanceDB stores embeddings for memory + document search
The broker protocol (version 5) correlates every request after the handshake with a reply id, bounds every wait (10 s for requests, 30 s for the handshake, 180 s in the stdio relay), runs all store access off the async workers (spawn_blocking, with a write gate for mutations), and deduplicates keyed posts for an hour. On Unix the daemon accepts connections before its store opens and answers only Ping until it does. After a backend restart the relay re-sends reads and keyed posts once and fails everything else with a retryable error. The detached daemon logs to daemon.log (rotated at 8 MiB). See Agent coordination and ADR-0013.
Why a daemon? Content-addressed blobs and a single Fjall writer are the key wins: all repos/worktrees on one machine share blob deduplication, and N sessions can read concurrently from a single index without downgrade-to-read-only issues. The daemon is the sole Fjall writer and enforces this via socket-bind-as-lock (singleton enforcement).
The daemon’s socket, lock, log and comms store live in a protocol-generation subdirectory of the comms directory (comms/g1/). A basemind binary only looks inside its own protocol generation, so two installed releases of different generations each run their own daemon instead of replacing each other’s. Releases up to 0.33 use the comms directory itself. The daemon registry is scoped the same way, so comms stop --all and comms doctor cover their own generation’s daemons only. The workspace index, memory and git-history stores are not per generation: a second daemon cannot serve a workspace the first one holds. A 0.33 plugin launcher still kills other versions’ processes started from the plugin cache, so the two coexist fully only once it is gone. With BASEMIND_ALLOW_HTTP, the second daemon to start binds an OS-assigned port instead of the default; read it from http.addr. See Upgrading.