Document Search

basemind extracts 90+ file formats (PDF, Office, HTML, email, images via OCR) into a LanceDB vector store and answers meaning-based queries with optional cross-encoder reranking. Web pages scraped or crawled into the same store are searchable the same way.

--features documents or --features full

Requirements

search_documents needs a build with --features documents (or full). The memory_* tools are a separate feature — --features memory (also in full). Without the relevant feature the tool dispatches but returns an error.
Web ingestion (web_scrape, web_crawl, web_map) needs --features crawl. When that feature is off these tools are not registered at all.
Documents must be scanned first: basemind scan with the documents feature extracts and embeds them into .basemind/.

Core tool

`search_documents`

Semantic search across PDFs, Office, HTML, email, images (OCR), and web pages. Returns chunk-level hits with path, matched text, byte span, vector distance, and — when enabled at scan time — cross-encoder rerank score, named entities, and document summary.

{
  "query": "how is the index schema versioned",
  "limit": 10,
  "entity_category": "PERSON",
  "keywords_contains": "release"
}

Filter results by:

entity_category — narrow to documents mentioning entities like PERSON, LOCATION, ORGANIZATION (NER-extracted at scan time).
keywords_contains — narrow to documents with a matching keyword (extracted at scan time).
mime_type — filter by file type, e.g. "application/pdf" or "text/html".

Each hit carries:

path — file path or web scope (web:<host>/<path>).
chunk_idx — which chunk within the document.
text — the matched passage.
byte_span — precise location in the source.
distance — L2 vector distance (lower = better).
rerank_score — cross-encoder score in [0, 1] (higher = better, only when reranking is enabled).
keywords / entities / summary — document-level metadata (when enabled at scan).

Web ingestion

`web_scrape`

Fetch and index a single page, adding it to the document store.

{
  "url": "https://docs.example.com/guide",
  "scope": "docs"
}

Results land tagged with a scope (default: web:<host>), making them queryable the same way as local documents.

`web_crawl`

Follow links from a seed URL up to a configurable depth, indexing all discovered pages.

{
  "url": "https://docs.example.com/",
  "max_depth": 2,
  "max_pages": 100
}

`web_map`

Discover a site’s pages without fetching bodies — returns a sitemap.

{
  "url": "https://docs.example.com"
}

Useful for “what pages exist on this site?” without the download cost.

Shared memory

Store key–value pairs that persist across sessions and are queryable by any agent on the same repo (determined by normalized git origin URL). Unrelated repos keep separate memory.

`memory_put`

Store a value.

{
  "key": "architecture_notes",
  "value": "The scanner uses rayon parallel iteration..."
}

`memory_get` / `memory_list`

Retrieve a value by exact key or list keys by prefix.

{
  "key": "architecture"
}

`memory_search`

Semantic search over stored values.

{
  "query": "how is parallelism managed",
  "limit": 5
}

Configuration

In .basemind/basemind.toml:

[documents]
enabled = true
embed = true  # set to false to disable embeddings (keyword search only)

[documents.reranker]
enabled = true  # cross-encoder reranking for document chunks

Discipline

Use search_documents instead of opening PDFs/Office/HTML by hand. Semantic search finds passages about a topic without keyword matching.
Use web_scrape / web_crawl to pull docs into the RAG store. Once indexed, they are searchable alongside local documents.
Use memory_put / memory_search for agent-generated insights. Store findings that you want to recall in future sessions.
Filter by entity_category or mime_type when the query is too broad. Narrow results to a document class.

Document Search

Requirements

Core tool

search_documents

Web ingestion

web_scrape

web_crawl

web_map