Skip to content

Document Search

basemind extracts PDFs, Office files, spreadsheets, email, HTML, XML, Markdown and Jupyter notebooks (and, with OCR, raster images and scanned PDFs) into a LanceDB vector store and answers meaning-based queries with optional cross-encoder reranking.

--features documents or --features full; OCR needs the documents-ocr feature and entity filtering needs documents-ner (both in full)

  • docs_search needs a build with --features documents (or full). memory_save and memory_search are gated on the separate --features memory (also in full). Without the relevant feature the tool is advertised but returns an isError result naming the Cargo feature.
  • OCR and named-entity recognition are separate Cargo features, documents-ocr and documents-ner. The published binaries (Homebrew, npm, pip, GitHub releases) are built with full and have both. A source build with --features documents alone has neither: raster images are not indexed as documents, a PDF with no text layer is indexed with no text and logs a warning naming the file and documents-ocr, [documents.ner] enabled = true logs a warning and has no effect, and a docs_search call with entity returns an error naming documents-ner.
  • Archives (.zip, .tar, .7z, .gz) are never extracted.
  • Documents must be scanned first: basemind scan with the documents feature extracts and embeds them into the global cache.

Semantic search across PDFs, Office, HTML, email, and images (OCR). Returns chunk-level hits with path, matched text, byte span, vector distance, and — when enabled at scan time — cross-encoder rerank score, named entities, and document summary. CLI: basemind memory documents.

{
"query": "how releases are versioned",
"limit": 10,
"entity": "PERSON",
"keywords": "release"
}

Filter results by:

  • entity — narrow to documents mentioning entities like PERSON, LOCATION, ORGANIZATION (NER-extracted at scan time). Needs a build with documents-ner; without it the call returns an error instead of an empty result.
  • keywords — narrow to documents with a matching keyword (extracted at scan time).
  • mime_type — filter by file type, e.g. "application/pdf" or "text/html".

Each hit carries:

  • path — file path.
  • chunk_idx — which chunk within the document.
  • text — the matched passage.
  • byte_span — precise location in the source.
  • distance — L2 vector distance (lower = better).
  • rerank_score — cross-encoder score in [0, 1] (higher = better, only when reranking is enabled).
  • keywords / entities / summary — document-level metadata (when enabled at scan).

Web ingestion (the web tool and basemind web) was removed. Pages an older version scraped or crawled stay in the document store under the scope web:<host> and are no longer refreshed. docs_search searches this repository’s scope by default, so pass scope: "web:<host>" (--scope web:<host> on the CLI) to search them.

Store key–value pairs that persist across sessions and are queryable by any agent on the same repo (determined by normalized git origin URL). Unrelated repos keep separate memory.

Store a note. Saving an existing key replaces it. CLI: basemind memory put.

{
"key": "decision/auth-tokens",
"value": "Tokens rotate every 24h.",
"tags": ["auth"]
}

Recall notes. query searches by meaning, key reads one note, prefix lists by key prefix, and no arguments lists everything. A listing pages by next_cursor; truncated with truncated_reason: "scan_cap" says its total stopped counting and is a lower bound. CLI: basemind memory search / get / list.

{
"query": "token rotation"
}
{
"key": "decision/auth-tokens"
}

Deleting a note is CLI only: basemind memory delete <key>.

In basemind.toml:

[documents]
enabled = true
embed = true # set to false to disable embeddings (keyword search only)
include = [] # globs selecting which non-code files become documents (empty = all that pass [scan])
exclude = [] # globs never indexed as documents (wins over include)
max_file_bytes = 52428800 # per-document size cap (50 MiB), independent of [scan] max_file_bytes
embed_include = [] # embed only matching documents (empty = all); embed_exclude always wins
[documents.reranker]
enabled = true # cross-encoder reranking for document chunks
  • Use docs_search instead of opening PDFs/Office/HTML by hand. Semantic search finds passages about a topic without keyword matching.
  • Use memory_save / memory_search for agent-generated insights. Store findings that you want to recall in future sessions.
  • Filter by entity or mime_type when the query is too broad. Narrow results to a document class.

Code intelligence · Git intelligence · Code search