When a fast dense retriever keeps only the top‑B pages for a late‑interaction reranker, a relevant page outside the top‑B can never be recovered. This kit measures how many human‑annotated relevant pages survive each stage of a dense → late‑interaction cascade for B ∈ {10, 25, 50, 100}, using exact dense search in Qdrant and server‑side MaxSim reranking with NeoMME‑260M‑Retriever on the ViDoRe v3 benchmark.
It is a pinned, reproducible measurement, not a new method or a general claim about reranking. See Limitations.
| Component | Version / revision | Role |
|---|---|---|
| Dataset | vidore/vidore_v3_computer_science @ d5cc75883d92e294f0c0fc2662551c9708a06ebc |
pages, queries, human qrels |
| Model | Hcompany/NeoMME-260M-Retriever @ 481083153ac22931afbd614e672212bffbcb666e |
one forward pass → dense (1024‑d) + multivector (128‑d / token) |
| Qdrant server | v1.19.2 (binary SHA256‑verified) |
exact dense search + MAX_SIM rerank |
qdrant-client |
1.19.1 |
client |
transformers |
5.19.0 |
model + processor (needs torchvision) |
sentence-transformers |
6.1.0 |
util.mean_maxsim parity reference |
datasets |
5.1.0 |
data loading |
| Precision | float32, no autocast | never switched silently |
| Resolution | longest side 1024 px (primary_1024) |
resource‑bounded preset; the model default is 2048 |
Torch / torchvision come from the Colab runtime and are recorded per run in run_manifest.json and pip_freeze.txt.
110 human‑written English queries, 545 judged pages, 1,360 pages. Macro means, 95 % bootstrap CIs over whole queries
(2,000 replicates, seed 42). Full tables, strata and limits: RESULTS.md.
| B | candidate recall | top‑10 recall after rerank | Δ vs dense order [95 % CI] |
|---|---|---|---|
| 10 | 0.583 | 0.583 | +0.000 (identical) |
| 25 | 0.709 | 0.646 | +0.063 [+0.030, +0.103] |
| 50 | 0.801 | 0.678 | +0.095 [+0.052, +0.141] |
| 100 | 0.893 | 0.715 | +0.132 [+0.084, +0.181] |
Dense order alone: 0.583. Unrestricted full‑corpus late‑interaction reference: 0.723. Reranking the top‑100 is not distinguishable from that reference here (Δ = −0.009 [−0.027, +0.011]). Coverage of all annotated pages in the top‑10 moves only from 0.31 to 0.36 and was not significant.
Cost on the T4: embedding 1,360 pages 328 s, Qdrant indexing 299 s, the whole rerank experiment 20 s; peak 2.3 GB VRAM, 3.7 GB host RAM, 0.64 GB Qdrant data. Qdrant vs torch MeanMaxSim agree to 1.4e‑7.
On computer_science (110 queries, 1,360 pages, 1024 px, float32):
- The candidate stage is the bottleneck. At B = 10 the reranker changes nothing (+0.000), because it can only reorder what the dense stage passed on.
- Gains grow with the budget: top‑10 recall rises by +0.063 (B = 25), +0.095 (B = 50) and +0.132 (B = 100) over dense order, with 95 % CIs excluding zero.
- A small budget gets close to the full search: reranking the dense top‑100 (0.715) is not statistically distinguishable from late‑interaction scoring of the whole corpus (0.723).
- Not shown: that this holds in other domains, at other resolutions or with other models, or that it improves answer quality. Those are open (see How to build on this).
A standard two‑stage retrieve‑then‑rerank cascade; nothing here is a new method.
-
Stage 1, dense candidates. Exact (non‑approximate) cosine search over the 1024‑d page vectors in Qdrant; keep the top‑B pages.
-
Stage 2, late‑interaction rerank. Server‑side
MAX_SIMover the 128‑d token vectors of only those B pages (ColBERT‑style MaxSim; seedocs/REFERENCES.md):MeanMaxSim(q, p) = (1 / |q|) · Σ_i max_j ⟨q_i, p_j⟩where
q_iare the valid query‑token vectors andp_jthe page‑token vectors. Qdrant returns the sum; the kit divides by|q|and checks it againstsentence_transformers.util.mean_maxsim. -
Measure. For every annotated page, record whether it was lost at stage 1 (
candidate_loss), lost by the rerank (reranking_loss) or kept in the top‑k (survived_top_k).
- Open the notebook with the badge above and select Runtime → Change runtime type → T4 GPU.
- Run top to bottom. If the install cell asks for a restart, restart and run again from the top.
- Defaults:
MODE="full",DATASET="computer_science",PRESET="primary_1024"; a smoke fixture runs first (8 queries / 256 pages, engineering check only). - Optional section 15 repeats the same protocol on
hr,physics,energy,pharmaceuticals(each pinned insrc/vesk/config.py) and aggregates them. It has not been run yet; no cross‑domain numbers exist.
Every stage checkpoints per item with checksums and resumes after a disconnect (USE_DRIVE=True keeps checkpoints on
Drive; that path is untested). Nothing is published, and no paid service or secret is needed.
pip install -e .
vesk full --root ./work --full-late # or: python -m vesk.cli full ...
python -m vesk.aggregate OUT_DIR run_a.zip run_b.zip # combine finished full runs
pytest tests # real pinned data, network required- Data gate. Pinned revision; asserts counts and invariants; only
language == english AND query_generator == human. The LLM‑mergedanswerand themarkdowncolumns are never used. - Embed once. A single float32 forward pass gives
embeddings(multivector) anddense_embeddings; per‑item checksummed shards; padding trimmed, all other positions kept. - Index. Qdrant collection with named vectors
dense(cosine) andlate(multivector, MAX_SIM,m=0, cold storage). - Candidates → rerank. Exact dense top‑B, then a server‑side
Prefetchrerank with the late vector; nested budgets share one dense ranking. - Parity. Qdrant sum ÷ query‑token count must match
sentence_transformers.util.mean_maxsimwith masks. - Metrics. Candidate recall / coverage, final top‑5/10, nDCG (gain 2^score − 1), score‑1 / score‑2 diagnostics, paired bootstrap over query IDs, strata with n, and a loss label per page (candidate loss, reranking loss, survived).
- Traces.
trace_viewer.htmlshows deterministic example queries with the human boxes drawn on the pages; images are fetched on demand from the pinned revision.
src/vesk/ pipeline (config, data, embed, qdrant_store, scoring, metrics, report, aggregate, cli)
notebooks/ Colab notebook (+ executed/ the v1 T4 run with outputs)
results/ verified outputs of the computer_science run (CSVs, plots, manifest, viewer)
docs/ PROTOCOL, LIMITATIONS, ATTRIBUTION, SOURCES, REFERENCES, BLOG_PROMPT
tests/ real-data tests (no synthetic fixtures)
tools/build_notebook.py regenerates the notebook from src/
Each item is open work; none has been run.
- Replicate across domains: run section 15 of the notebook (
hr,physics,energy,pharmaceuticals) andpython -m vesk.aggregate; check whether "top‑100 ≈ full late" holds. - Resolution: run the 2048 px preset (the model default) and compare against 1024 px.
- Swap the candidate generator: replace the dense stage with BM25 or a hybrid, or the reranker with another multivector model, keeping the same metrics and bootstrap.
- Diagnose failures: use the per‑page loss labels and
trace_viewer.htmlto see which pages the dense stage loses and which the rerank loses. - Beyond page recall: connect survived pages to answer quality; this kit measures no answers.
- One domain (two OpenStax textbooks), one model, one resolution, 110 queries; no general claim about reranking.
- Judgments are incomplete: unjudged pages are not negatives, so recall is a lower bound.
- Top‑k ceiling: 10 of 110 queries have more than 10 annotated pages, so best possible top‑10 recall is ≈ 0.974.
- No answer‑quality measurement. 1024 px is not a reproduction of published scores.
- Not run yet: the 2048 px preset, Drive resume, and the cross‑domain section.
Code: MIT. ViDoRe v3 annotations: CC BY 4.0. Model: Apache 2.0. Page images are third‑party OpenStax material and are
not redistributed here. Details in docs/ATTRIBUTION.md; verified sources in
docs/SOURCES.md; reference papers in docs/REFERENCES.md.
If you use the data or model, cite ViDoRe V3 (arXiv:2601.08620) and NeoMME (arXiv:2609.01657). For this kit, cite the repository URL and the commit hash you ran.