A local-first benchmark for evaluating retrieval quality inside agentic workflows across tools, memories, and grounding evidence.
Classic RAG evaluation asks whether a system retrieved the right document. Agentic systems also need to select the right tool, apply the right durable memory or policy, and retrieve the right evidence before acting. This benchmark makes those retrieval targets measurable with Qdrant, FastEmbed, a small synthetic dataset, and reproducible local commands.
Current status: runnable local benchmark with validated dense, sparse, rerank, and hybrid-rerank strategies.
The recommended path uses Qdrant local-memory mode, which does not require Docker.
Requires Python 3.10+. Using a virtual environment is recommended.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
python -m pytest
python -m app.evaluate --strategy dense --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.mdThe benchmark is local-first, but first-run FastEmbed model downloads may require network access unless the required model files are already cached.
Document-only retrieval can hide failures that matter in agentic systems. A workflow may find a relevant document while selecting the wrong tool, skipping a required policy memory, or grounding the final answer in weak evidence.
This project isolates those retrieval decisions so quality and latency trade-offs are visible without building a full agent framework.
| Retrieval surface | Question answered | Example target |
|---|---|---|
| Tool | Did the system select the right callable capability? | tax_validator |
| Memory | Did it retrieve the right policy, preference, or durable instruction? | human_approval_policy |
| Evidence | Did it retrieve the right grounding document or note? | invoice_tax_rules |
Core metrics:
Tool@1Tool@3Memory@1Evidence@3- mean latency
- P95 latency
- gain versus dense baseline where applicable
| Strategy | Description | Expected trade-off |
|---|---|---|
dense |
FastEmbed dense embeddings searched in Qdrant | Strong semantic baseline |
dense-rerank |
Dense candidates reranked with a FastEmbed cross-encoder | Better top-rank quality, higher latency |
sparse |
FastEmbed sparse BM25-style vectors searched in Qdrant | Very fast exact-term matching |
hybrid-rerank |
Dense and sparse candidates merged, deduplicated, then reranked | Strong quality, more moving parts |
Final release-readiness rerun in Qdrant local-memory mode:
| Strategy | Tool@1 | Tool@3 | Memory@1 | Evidence@3 | Mean ms | P95 ms |
|---|---|---|---|---|---|---|
| dense | 0.700 | 0.900 | 0.650 | 0.550 | 61.46 | 83.96 |
| dense-rerank | 0.750 | 0.850 | 0.700 | 0.750 | 1123.66 | 1235.73 |
| sparse | 0.550 | 0.800 | 0.600 | 0.750 | 1.52 | 2.05 |
| hybrid-rerank | 0.750 | 0.850 | 0.700 | 0.750 | 1099.49 | 1322.92 |
Latency is run- and environment-sensitive. The values above are the final verified local-memory rerun for this repository state.
- Dense gives the strongest Tool@3 with moderate latency.
- Sparse is extremely fast and improves Evidence@3, but hurts Tool@1.
- Reranking improves Tool@1, Memory@1, and Evidence@3, but adds significant latency.
- Hybrid-rerank matches rerank quality in this run, while still remaining latency-heavy.
- Evidence retrieval remains below the original demonstration target, which is a useful failure-mode signal rather than something to tune away by changing labels.
python -m app.evaluate --strategy dense --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.md
python -m app.evaluate --strategy dense-rerank --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.md
python -m app.evaluate --strategy sparse --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.md
python -m app.evaluate --strategy hybrid-rerank --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.mdOptional rerank candidate count:
python -m app.evaluate --strategy dense-rerank --candidate-k 10 --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.mdThe benchmark loads JSONL data, validates ground-truth target ids, embeds retrievable items with FastEmbed, creates Qdrant collections, retrieves candidates for each surface, optionally reranks them, computes metrics, and writes a report.
See Architecture for the detailed component map and execution flow.
local-memory is the default path because it avoids Docker and server setup:
python -m app.evaluate --strategy dense --qdrant-mode local-memory --recreate-collections --write-report reports/agentic_retrieval_results.mdDocker/server mode is optional:
docker compose up -d
python -m app.evaluate --strategy dense --qdrant-mode server --qdrant-url http://localhost:6333 --recreate-collections --write-report reports/agentic_retrieval_results.mdRestricted networks may block Docker Hub or related image-pull infrastructure. The benchmark is local-first, but first-run FastEmbed model downloads may require network access unless the required model files are already cached. Fully offline execution requires dependencies and model files to already be available locally.
Default models:
- dense:
BAAI/bge-small-en-v1.5 - sparse:
Qdrant/bm25 - reranker:
Xenova/ms-marco-MiniLM-L-6-v2
This benchmark does not require OpenAI, Anthropic, external LLM APIs, paid APIs, or cloud services.
- Architecture
- Results interpretation
- Offline and restricted network notes
- Future PR plan
- Benchmark report
The upstream discussion draft is kept locally for human review and is not part of the public repository.
| This is | This is not |
|---|---|
| A local benchmark for agentic retrieval surfaces | A full agent framework |
| A synthetic evaluation harness | A scientific leaderboard |
| A Qdrant/FastEmbed example candidate | An official Qdrant project |
| A no-external-LLM benchmark | An OpenAI/Anthropic app |
| A small methodology artifact for review | A claim that one strategy is generally best |
- The dataset is small and synthetic.
- Results are methodology/demo results, not scientific or leaderboard claims.
- Latency varies by hardware, run order, model loading, and cache state.
- FastEmbed model downloads may occur on first run.
- Evidence retrieval still needs future analysis because the best verified Evidence@3 is 0.750, below the original demonstration target.
MIT License. See LICENSE.
The first upstream step should be a Qdrant/FastEmbed GitHub Discussion, not a large unsolicited PR. The discussion should ask whether maintainers would find this useful as a docs example or experimental benchmark and where they would prefer it to live.
The intended upstream contribution would likely be a trimmed docs/example or experiments artifact, not this full repository.
This repository is an independent benchmark project using Qdrant and FastEmbed. It is not official, endorsed, or approved by Qdrant/FastEmbed maintainers unless they explicitly say so in the future.