Repository navigation
Abstention: one reranker threshold discards 16/16 paraphrase answers it already retrieved #113
Description
On memory-bench v0.1 the vector row says "I don't know" on 18 of 69 answerable questions; 16 of those 18 are paraphrase (10/10) and paraphrase_bridge (6/6). Retrieval already found most of them (paraphrase R@5 0.80), but the abstention rule (top reranker score < 0) throws the answer away: the mmarco cross-encoder scores zero-word-overlap hits below zero (median -4.8, see bench/README.md "Baseline").
One threshold is doing two jobs: rejecting absent entities and judging paraphrase relevance.
Proposal: keep the reranker gate for lexical matches, and for queries with no lexical overlap with the top hit use the bi-encoder cosine (or a separately calibrated threshold) as the abstention signal. Acceptance: false_abstain on paraphrase drops well below 1.00 without abstain_correct falling below 0.45 on the 22 absent questions. Numbers from bench/results/sqlite-graph-memory-vector.json.
Activity
tonydzi commentedon Nov 11, 2026
Correction to the diagnosis above: the threshold is not the only cause. The adapter also returns nothing when the shared extractor finds no sentence with an overlapping word (
if top < ABSTAIN_BELOW or not sent). Rerunning_extract.best_sentenceon the saved rankings inresults/sqlite-graph-memory-vector.jsonwith the score gate removed still leaves 6 of the 16 paraphrase questions unanswered. Also, onparaphrase_bridgeretrieval itself mostly misses (R@5 0.08), so those 6 refusals follow a retrieval miss, not a rejected hit.So the first step is to split the 16 into retrieval misses, threshold rejections and extractor misses. Any replacement rule must not depend on the bench's class labels, and gets frozen before it is scored on a freshly generated vault.
tonydzi commentedon Nov 11, 2026
Measured the threshold's effect on the saved rankings (replaying
_extract.best_sentencereproduces 91/91 saved answers first):So the gate buys 7 correct refusals for 12 false refusals and 2 good answers. Separately, the shared extractor returns nothing on 10/10
paraphrasequestions even when given the gold support notes, so paraphrase silence is mostly the extractor floor, not retrieval. Any fix here should be scored against both baselines.