Architecture

basemind is a single Rust crate with one binary (basemind) serving two roles: basemind scan indexes a workspace, and basemind serve runs the MCP stdio server.

High-Level Pipeline

Your project (code + documents + git history)
  ↓
basemind scan
  ├─ Tree-sitter parser pool (300+ languages)
  ├─ Extract L1 (outlines: symbols, signatures, imports)
  ├─ Extract L2 (calls: callee, byte offset, line/col) [if eager_l2=true]
  ├─ Extract L3 (structural hash of symbol bodies)
  └─ Write to local cache
      ├─ Content-addressed msgpack blobs (.basemind/blobs/)
      └─ Fjall LSM index (.basemind/views/<view>/index.fjall/)
  ↓
basemind serve
  ├─ Load index + blobs into memory
  ├─ Answer MCP tool requests instantly
  └─ Watch filesystem for changes (auto-rescan)

Storage: Blobs + Index

basemind persists two complementary forms of your project:

Content-Addressed Msgpack Blobs

Location: .basemind/blobs/<hash>.{l1,l2,l3}.msgpack

Each extraction tier is a separate msgpack blob, keyed by content hash. This enables deduplication across files with identical bodies.

• L1 (always): outlines with symbols, signatures, imports, doc comments
• L2 (if eager_l2=true): call sites (callee name, byte offset, line/column)
• L3: structural hash of each symbol’s body (used by symbol_history)

Blobs are immutable and content-addressed, so they can be safely shared across multiple views (scans) and repos.

Fjall LSM Inverted Index

Location: .basemind/views/<view>/index.fjall/

A disk-backed, append-only LSM tree (Fjall) provides lightning-fast lookups:

Keyspace	Purpose
meta	Schema version and constants
symbols_by_path	Per-file symbol lookup
symbols_by_name	Name-prefix range scans (drives search_symbols)
calls_by_path	Per-file call lookup
calls_by_callee	Callee-prefix range scans (drives find_references)
imports_by_path	Per-file import lookup
imports_by_module	Module-prefix range scans (drives dependents)
implementations_by_path	Per-file implementation lookup
implementations_by_trait	Trait-prefix range scans (drives find_implementations)
embeddings	Reserved for future in-Fjall vector index
memory_by_key	Agent memory namespaced by (scope, visibility, owner)

All composite keys are length-prefixed (u16:len ‖ bytes), ensuring a prefix like Foo never spills into Foobar.

Schema version is stamped in meta. On version mismatch, the next basemind scan auto-wipes and rebuilds.

LanceDB Vector Store

Location: .basemind/lancedb/

Powers semantic search (code search, document search, memory search). Vectors are indexed for fast KNN queries; on-disk format is LanceDB’s columnar structure.

Scan Pipeline

basemind scan and basemind watch (incremental) share the same per-file pipeline:

Walker (gitignore-aware) → filter by glob + size cap
  ↓
rayon par_iter (parallel across all CPU cores)
  ↓
process_file(rel_path, file_contents):
  1. lang::detect()              ← tree-sitter language pack
  2. Extract L1 (outline)        ← tree-sitter queries
  3. Extract L2 (calls)          ← if eager_l2=true
  4. Write L1 to blob store
  5. Write L2 to blob store      ← if eager_l2=true
  6. Index lookup:
     a. Read existing entries (if re-scan)
     b. Stage all deletes + inserts in one atomic Fjall batch
  7. Per-file commit             ← Fjall handles cross-thread locking
  ↓
collect FileResult { rel, l1_hash, l2_hash?, … }
  ↓
apply_outcomes:
  · Write Index meta
  · Remove stale files via IndexWriter::remove_file

Key Invariants

Per-file commit — Every file commits its Fjall batch before returning. Fjall handles cross-thread locking; the scanner does not.

Atomic upsert — IndexWriter::upsert_file is read-before-write: it reads existing entries first to derive secondary-index keys for deletion, then stages all deletes + inserts in one batch. This prevents torn state on re-scan.

Eager L2 cost — Eager L2 (call-site extraction) is on by default and adds to scan time; the 81k-file TypeScript scan lands around 18s. Set eager_l2 = false to skip it for a faster scan, trading away reference search.

scan_paths removal mirror — When a file disappears between scans, scan_paths calls IndexWriter::remove_file so secondary indexes don’t leak stale entries.

Extraction Tiers

L1: Outlines (Always)

Per-file symbol table with signatures, line/column locations, import statements, and doc comments. Generated by tree-sitter queries (hand-written overrides at src/queries/<language>.scm, falling back to tree-sitter-language-pack’s tags.scm).

Used by: outline, search_symbols, goto_definition, call-graph traversal.

L2: Call Sites (Optional, Default On)

Every call site in a file: callee name, byte offset, line/column. Extracted only if eager_l2 = true.

Used by: find_references, find_callers, call_graph.

Disabling it speeds up the scan but makes reference search unavailable.

L3: Structural Hashes

Cryptographic hash of each symbol’s body. Used to detect when a symbol’s implementation changed across commits.

Used by: symbol_history.

Document Extraction (xberg)

When [documents] enabled = true, scanner_docs.rs runs a separate pass:

• Reads PDFs, Office files, HTML, images, email
• Extracts text (with OCR for images)
• Chunks and embeds via LanceDB
• Indexed by search_documents

Language Support

basemind uses the tree-sitter language pack (TSLP) to detect and parse 300+ languages. Language detection is dynamic:

File extension → tree_sitter_language_pack::detect_language
  ↓
LangId = &'static str (the pack name, e.g. "rust", "python", "typescript")
  ↓
Parser from pool (lazy init, reused across files)
  ↓
Query (hand-written override OR TSLP tags.scm fallback)
  ↓
Extract symbols + calls

All languages with a tags.scm query in TSLP (~100) work out of the box. For richer extraction, add a .scm override at src/queries/<pack-name>.scm.

Git Layer

basemind git commands are powered by gix (a pure-Rust Git implementation) with a git-history index for microsecond latency:

Git-History Index

Location: .basemind/git-cache/

Two-tier caching:

• In-process LRU (1024 entries per category by default; tune via basemind serve --git-cache-mem)
• Disk store (SHA-keyed):
  • commit_files/<sha>.msgpack — file list for a commit (immutable, never stales)
  • log/<head_sha>__<scope>.msgpack — commit log (keyed by HEAD, rolls off naturally)
  • blame/<sha>__<path_hash>.msgpack — blame data (immutable, never stales)

History queries (commits_touching, recent_changes, blame_*) are posting-list lookups over this index, answering in tens of microseconds flat.

The index is a pure accelerator: it’s used only when fresh (last_indexed_head == HEAD) and otherwise walks history directly, so it can never serve stale results. It rebuilds automatically on history rewrite (filter-repo, rebase, force-push).

Schema Versioning

Two persistent schemas, both tracking RELEASE_MINOR from src/version.rs:

• INDEX_SCHEMA_VER in src/index/mod.rs — Fjall keyspace format
• SCHEMA_VER in src/extract/mod.rs — msgpack blob format

Versioning Rules

Patch release (0.1.0 → 0.1.1): RELEASE_MINOR unchanged. Blobs + index must stay compatible — no format changes.

Minor release (0.1.x → 0.2.0): RELEASE_MINOR increments. On next basemind scan, .basemind/ is auto-wiped and rebuilt from source. Users are notified in release notes.

Wipe-on-mismatch is the explicit migration story. No manual steps — basemind scan handles it.

Directory Layout

.basemind/
├── basemind.toml           # Configuration
├── agent-id                # Persistent agent identity
├── blobs/
│   ├── abc123….l1.msgpack  # Outline: symbols, signatures, imports
│   ├── abc123….l2.msgpack  # Calls (if eager_l2=true)
│   └── abc123….l3.msgpack  # Structural hashes
├── views/
│   └── <view>/
│       └── index.fjall/    # Fjall LSM index
├── git-cache/
│   ├── commit_files/       # Cached file lists per commit
│   ├── log/                # Cached logs
│   └── blame/              # Cached blame data
└── lancedb/                # Vector store (documents, memory, code search)

MCP Surface

basemind serve exposes a stdio MCP server. All paths are byte-precise repo-relative (RelPath).

Tool conventions:

• Paths: always RelPath (repo-relative), never arbitrary strings
• Lists: capped at 1000 items (default 100); responses include any_truncated flag and cursor for pagination
• Index scans: internally use scan_cap = limit * 8 to bound work on common names
• Descriptions: state matching semantics (substring vs. prefix, scope-aware vs. name-only)
• Stability: new fields are additive with #[serde(default)]

Tool bodies live in src/mcp/helpers*.rs (sliced by area: helpers_documents.rs, helpers_calls.rs, helpers_graph.rs, etc.); src/mcp/tools*.rs contain only thin #[tool] shims.

Agent Comms & Shared Memory

A separate daemon process (singleton per user) owns a second Fjall store for agent communication:

• Rooms: per-repo, per-workspace, or global scope; agents auto-join based on scope chain
• Messages: front-matter stored in messages_by_room, bodies in message_body (fetched on demand for efficiency)
• Memory: namespaced by (scope, visibility, owner) — group memory is shared, individual memory is private to one agent
• Vectors: LanceDB stores embeddings for memory + document search

Why a separate daemon? Each basemind serve takes an exclusive flock on .basemind/, so comms can’t live there. The daemon is user-global and enforces singleton via socket-bind-as-lock.

Note

The architecture trades off simplicity for performance: content-addressed blobs enable deduplication, the inverted index gives instant lookups, and the git-history index makes history queries microsecond-fast. At the same time, incremental scanning and a split memory substrate support multi-agent coordination without blocking.