[Feature]: Multi-Query and HyDE (Hypothetical Document Embeddings) fusion retriever #215

Problem

In production RAG applications, single-query vector retrieval frequently suffers from vocabulary mismatch and underspecified user queries (e.g., short, colloquial, or poorly formulated search queries that miss domain-specific keywords).

Two established retrieval patterns solve this problem:

1. Multi-Query Expansion: An LLM generates 3–5 diverse semantic reformulations of the original query, which are executed in parallel and combined via Reciprocal Rank Fusion (RRF).
2. HyDE (Hypothetical Document Embeddings): An LLM generates a hypothetical answer passage. Embedding this hypothetical document retrieves actual documents closer in semantic space than the raw question alone.

Currently, developers building on Dynavec have to orchestrate multi-query loops and RRF fusion manually in application code.

Proposed solution

Implement first-class MultiQueryRetriever and HyDERetriever adapters in dynavec.retrieval (or dynavec.integrations):

1. MultiQueryRetriever

• Accepts any standard callable llm_generate_queries: Callable[[str], list[str]].
• Executes parallel vector searches across Dynavec for all query variations.
• Merges candidate result lists using Dynavec's built-in reciprocal_rank_fusion(result_lists, k=rrf_k, weights=weights).

from dynavec.retrieval import MultiQueryRetriever

retriever = MultiQueryRetriever(
    source=db.namespace("docs"),
    llm_generate_queries=lambda q: llm.generate_variations(q, n=3),
    top_k=4,
    rrf_k=60,
)

results = retriever.search("how does serverless vector storage work?")

2. HyDERetriever

• Accepts any callable llm_generate_hypothetical: Callable[[str], str].
• Generates a hypothetical answer, embeds it, and searches Dynavec.
• Optionally fuses the hypothetical search results with the raw user query search using RRF (include_original_query=True).

from dynavec.retrieval import HyDERetriever

hyde = HyDERetriever(
    source=db,
    llm_generate_hypothetical=lambda q: llm.generate_answer(q),
    top_k=4,
    include_original_query=True,
)

results = hyde.search("explain dynamo db storage pricing breakdown")

Alternatives considered

• Relying on external orchestrators (LangChain/LlamaIndex): While they have multi-query modules, having native, lightweight adapters in Dynavec avoids heavy third-party dependencies and leverages Dynavec's native RRF and namespace features directly.
• Single-query search with cross-encoder reranking: Cross-encoders improve reranking of existing candidates, but cannot retrieve documents that were completely missed in the initial candidate recall stage. Multi-Query/HyDE expands the candidate pool itself.

Additional context

• Zero heavy dependencies required (pure Python + standard callable LLM wrappers).
• Can easily be tested 100% offline using mock LLM generator functions in unit tests.
• I am happy to take this up and implement it with full unit tests and a runnable example script!