Document Search
basemind extracts PDFs, Office files, spreadsheets, email, HTML, XML, Markdown and Jupyter notebooks (and, with OCR, raster images and scanned PDFs) into a LanceDB vector store and answers meaning-based queries with optional cross-encoder reranking.
--features documents or --features full; OCR needs the documents-ocr
feature and entity filtering needs documents-ner (both in full)
Requirements
Section titled “Requirements”docs_searchneeds a build with--features documents(orfull).memory_saveandmemory_searchare gated on the separate--features memory(also infull). Without the relevant feature the tool is advertised but returns anisErrorresult naming the Cargo feature.- OCR and named-entity recognition are separate Cargo features,
documents-ocranddocuments-ner. The published binaries (Homebrew, npm, pip, GitHub releases) are built withfulland have both. A source build with--features documentsalone has neither: raster images are not indexed as documents, a PDF with no text layer is indexed with no text and logs a warning naming the file anddocuments-ocr,[documents.ner] enabled = truelogs a warning and has no effect, and adocs_searchcall withentityreturns an error namingdocuments-ner. - Archives (
.zip,.tar,.7z,.gz) are never extracted. - Documents must be scanned first:
basemind scanwith the documents feature extracts and embeds them into the global cache.
Core tool
Section titled “Core tool”docs_search
Section titled “docs_search”Semantic search across PDFs, Office, HTML, email, and images (OCR). Returns
chunk-level hits with path, matched text, byte span, vector distance, and — when enabled
at scan time — cross-encoder rerank score, named entities, and document summary. CLI:
basemind memory documents.
{ "query": "how releases are versioned", "limit": 10, "entity": "PERSON", "keywords": "release"}Filter results by:
entity— narrow to documents mentioning entities likePERSON,LOCATION,ORGANIZATION(NER-extracted at scan time). Needs a build withdocuments-ner; without it the call returns an error instead of an empty result.keywords— narrow to documents with a matching keyword (extracted at scan time).mime_type— filter by file type, e.g."application/pdf"or"text/html".
Each hit carries:
path— file path.chunk_idx— which chunk within the document.text— the matched passage.byte_span— precise location in the source.distance— L2 vector distance (lower = better).rerank_score— cross-encoder score in[0, 1](higher = better, only when reranking is enabled).keywords/entities/summary— document-level metadata (when enabled at scan).
Pages ingested by older versions
Section titled “Pages ingested by older versions”Web ingestion (the web tool and basemind web) was removed. Pages an older version scraped or
crawled stay in the document store under the scope web:<host> and are no longer refreshed.
docs_search searches this repository’s scope by default, so pass scope: "web:<host>"
(--scope web:<host> on the CLI) to search them.
Shared memory
Section titled “Shared memory”Store key–value pairs that persist across sessions and are queryable by any agent on the same repo (determined by normalized git origin URL). Unrelated repos keep separate memory.
memory_save
Section titled “memory_save”Store a note. Saving an existing key replaces it. CLI: basemind memory put.
{ "key": "decision/auth-tokens", "value": "Tokens rotate every 24h.", "tags": ["auth"]}memory_search
Section titled “memory_search”Recall notes. query searches by meaning, key reads one note, prefix lists by key prefix, and
no arguments lists everything. A listing pages by next_cursor; truncated with
truncated_reason: "scan_cap" says its total stopped counting and is a lower bound. CLI:
basemind memory search / get / list.
{ "query": "token rotation"}{ "key": "decision/auth-tokens"}Deleting a note is CLI only: basemind memory delete <key>.
Configuration
Section titled “Configuration”In basemind.toml:
[documents]enabled = trueembed = true # set to false to disable embeddings (keyword search only)include = [] # globs selecting which non-code files become documents (empty = all that pass [scan])exclude = [] # globs never indexed as documents (wins over include)max_file_bytes = 52428800 # per-document size cap (50 MiB), independent of [scan] max_file_bytesembed_include = [] # embed only matching documents (empty = all); embed_exclude always wins
[documents.reranker]enabled = true # cross-encoder reranking for document chunksDiscipline
Section titled “Discipline”- Use
docs_searchinstead of opening PDFs/Office/HTML by hand. Semantic search finds passages about a topic without keyword matching. - Use
memory_save/memory_searchfor agent-generated insights. Store findings that you want to recall in future sessions. - Filter by
entityormime_typewhen the query is too broad. Narrow results to a document class.