Document Search

basemind extracts 90+ file formats (PDF, Office, HTML, email, images via OCR) into a LanceDB vector store and answers meaning-based queries with optional cross-encoder reranking. Web pages scraped or crawled into the same store are searchable the same way.

--features documents or --features full

Requirements

• The memory MCP tool’s documents mode needs a build with --features documents (or full). The other memory modes (put, get, list, search, delete, audit) are gated on the separate --features memory (also in full). Without the relevant feature the tool is advertised but the mode’s body returns an error.
• Web ingestion (web modes scrape, crawl, map) needs --features crawl. When that feature is off the web tool is not registered at all.
• Documents must be scanned first: basemind scan with the documents feature extracts and embeds them into .basemind/.

Core tool

memory mode documents

Semantic search across PDFs, Office, HTML, email, images (OCR), and web pages. Returns chunk-level hits with path, matched text, byte span, vector distance, and — when enabled at scan time — cross-encoder rerank score, named entities, and document summary. CLI: basemind memory documents.

{
  "mode": "documents",
  "query": "how is the index schema versioned",
  "limit": 10,
  "entity_category": "PERSON",
  "keywords_contains": "release"
}

Filter results by:

• entity_category — narrow to documents mentioning entities like PERSON, LOCATION, ORGANIZATION (NER-extracted at scan time).
• keywords_contains — narrow to documents with a matching keyword (extracted at scan time).
• mime_type — filter by file type, e.g. "application/pdf" or "text/html".

Each hit carries:

• path — file path or web scope (web:<host>/<path>).
• chunk_idx — which chunk within the document.
• text — the matched passage.
• byte_span — precise location in the source.
• distance — L2 vector distance (lower = better).
• rerank_score — cross-encoder score in [0, 1] (higher = better, only when reranking is enabled).
• keywords / entities / summary — document-level metadata (when enabled at scan).

Web ingestion

web mode scrape

Fetch and index a single page, adding it to the document store. CLI: basemind web scrape.

{
  "mode": "scrape",
  "url": "https://docs.example.com/guide",
  "scope": "docs"
}

Results land tagged with a scope (default: web:<host>), making them queryable the same way as local documents.

web mode crawl

Follow links from a seed URL up to a configurable depth, indexing all discovered pages. CLI: basemind web crawl.

{
  "mode": "crawl",
  "url": "https://docs.example.com/",
  "max_depth": 2,
  "max_pages": 100
}

Note

The crawler honors robots.txt by default (disable with config). It SSRF-blocks private and loopback hosts unless [crawl].allow_private_network = true is set in .basemind/basemind.toml.

web mode map

Discover a site’s pages without fetching bodies — returns a sitemap. CLI: basemind web map.

{
  "mode": "map",
  "url": "https://docs.example.com"
}

Useful for “what pages exist on this site?” without the download cost.

Shared memory

Store key–value pairs that persist across sessions and are queryable by any agent on the same repo (determined by normalized git origin URL). Unrelated repos keep separate memory.

memory mode put

Store a value. CLI: basemind memory put.

{
  "mode": "put",
  "key": "architecture_notes",
  "value": "The scanner uses rayon parallel iteration..."
}

memory mode get / list

Retrieve a value by exact key or list keys by prefix. CLI: basemind memory get / basemind memory list.

{
  "mode": "get",
  "key": "architecture"
}

memory mode search

Semantic search over stored values. CLI: basemind memory search.

{
  "mode": "search",
  "query": "how is parallelism managed",
  "limit": 5
}

Configuration

In .basemind/basemind.toml:

[documents]
enabled = true
embed = true  # set to false to disable embeddings (keyword search only)

[documents.reranker]
enabled = true  # cross-encoder reranking for document chunks

Discipline

• Use memory mode documents instead of opening PDFs/Office/HTML by hand. Semantic search finds passages about a topic without keyword matching.
• Use web modes scrape / crawl to pull docs into the RAG store. Once indexed, they are searchable alongside local documents.
• Use memory modes put / search for agent-generated insights. Store findings that you want to recall in future sessions.
• Filter by entity_category or mime_type when the query is too broad. Narrow results to a document class.

Note

All results are paginated. Use next_cursor from one response as cursor in the next to fetch additional pages.

See also

Code intelligence · Git intelligence · Code search