Skip to content
This repository was archived by the owner on Oct 20, 2026. It is now read-only.

fix(search): stop a repeated URL from faking cross-engine agreement - #431

Merged
leookun merged 1 commit into
leookun:mainfrom
MaxFreedomPollard:fix/web-search-duplicate-url-rrf
Sep 15, 2026
Merged

fix(search): stop a repeated URL from faking cross-engine agreement#431
leookun merged 1 commit into
leookun:mainfrom
MaxFreedomPollard:fix/web-search-duplicate-url-rrf

Conversation

@MaxFreedomPollard

@MaxFreedomPollard MaxFreedomPollard commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Problem

WebSearch::search queries the 23 engines in catalog.rs in parallel and fuses their result lists with Reciprocal Rank Fusion. RRF exists to reward a document that several independent lists agree on, so it adds one 1 / (k + rank) term per list. merge (server/src/search/federation.rs:120) added a term per result instead:

match merged.get_mut(&result.url) {
    Some(existing) => {
        existing.score += score;                    // every repeat, the same engine's included
        if !existing.engines.contains(&engine) {    // the label is guarded, the score is not
            existing.engines.push(engine);
        }

An engine that lists one canonical URL at ranks 1 and 2 therefore scores 1/61 + 1/62 = 0.03252, which is what two independent engines both ranking it first are worth (2/61 = 0.03279). The engines field still correctly lists one engine, so the inflated score carries no sign of where it came from, and the score is what results.sort_by orders by at federation.rs:102.

Why an engine repeats a URL

The repeats come from this repository's own canonicalization rather than from odd input. canonicalize (server/src/search/engine.rs:295) drops the fragment, the trailing slash and the utm_*, gclid, fbclid, mc_cid and mc_eid parameters, and redirected_target (engine.rs:318) unwraps the Bing /ck/a, DuckDuckGo uddg and Google /url redirector links. Any two rows on one result page whose links differ only in one of those ways collapse onto one string. Output of canonical_url and result_url on current main:

https://doc.rust-lang.org/std/vec/struct.Vec.html#method.push   ->  https://doc.rust-lang.org/std/vec/struct.Vec.html
https://doc.rust-lang.org/std/vec/struct.Vec.html#method.pop    ->  https://doc.rust-lang.org/std/vec/struct.Vec.html
https://example.com/docs/                                       ->  https://example.com/docs
https://example.com/docs                                        ->  https://example.com/docs
https://example.com/p?utm_source=a                              ->  https://example.com/p
https://example.com/p?utm_source=b                              ->  https://example.com/p
//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.com%2Fp&rut=aaa  ->  https://example.com/p
//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.com%2Fp&rut=bbb  ->  https://example.com/p

Nothing removes the duplicate before it is scored. parse takes the first ten hits and returns them as they came (engine.rs:118 for HTML, :190 for JSON), and search hands the whole list to merge (federation.rs:83).

Fix

Put the score behind the guard that was already there for the engine label, so each engine contributes its best rank once. Cross-engine merging is untouched, and a repeat can still upgrade the stored snippet, which is the one thing a second row genuinely adds.

I did not dedupe inside parse instead: that would need the same bookkeeping in the HTML and JSON parsers, and it would throw away the later row's longer snippet. merge is where a URL becomes one entry, so it is where a list should count once.

Verification

# before (fix reverted, tests kept)
$ cargo test -p cursor-server --lib search::federation
test two_engines_that_agree_on_a_url_still_sum_both_ranks ... ok
test agreement_between_engines_outranks_one_engine_repeating_itself ... FAILED
test an_engine_that_lists_one_url_twice_contributes_a_single_rrf_term ... FAILED

---- an_engine_that_lists_one_url_twice_contributes_a_single_rrf_term stdout ----
assertion `left == right` failed
  left: 0.03252247488101534
 right: 0.01639344262295082

---- agreement_between_engines_outranks_one_engine_repeating_itself stdout ----
assertion failed: merged["https://example.com/agreed"].score >
    merged["https://example.com/repeated"].score

test result: FAILED. 1 passed; 2 failed

# after
$ cargo test -p cursor-server --lib search::federation
test result: ok. 3 passed; 0 failed

two_engines_that_agree_on_a_url_still_sum_both_ranks passes before and after; it is there to pin that a genuine two-engine merge still sums both ranks and keeps both labels. federation.rs had no tests before this, so the new module is at the end of the file.

Gates, on the toolchain CI uses (dtolnay/rust-toolchain@stable, here 1.98.0):

$ cargo fmt --all -- --check                                                   # clean
$ cargo clippy -p cursor-server --all-targets -- -D warnings                   # clean
$ cargo test  -p cursor-server --all-targets                                   # 152 lib tests + 14 integration suites, 0 failures
$ cargo clippy -p semble-core -p semble-benchmark --all-targets -- -D warnings # clean
$ cargo test  -p semble-core -p semble-benchmark --all-targets                 # 47 passed, 0 failed

@MaxFreedomPollard
fix(search): stop a repeated URL from faking cross-engine agreement
ff9978f 
`WebSearch::search` fuses the engines with Reciprocal Rank Fusion, which
sums one term per ranked list. `merge` added a term for every result
instead, so an engine that listed the same canonical URL twice had both
of its ranks counted:

    1/61 + 1/62 = 0.0325

That is what two independent engines agreeing at rank 1 are worth
(2/61 = 0.0328), produced from a single list. Agreement across engines is
the only ranking signal this federation has, and a repeat inside one
engine forges it.

The repeats come from the repository's own canonicalization, not from
exotic input. `canonicalize` in `search/engine.rs` drops the fragment,
the trailing slash and the `utm_*`, `gclid`, `fbclid` and `mc_*`
parameters, and `redirected_target` unwraps the Bing, DuckDuckGo and
Google redirector links, so rows that are visibly distinct on one result
page collapse onto one URL. Nothing dedupes an engine's own list before
`merge` sees it.

`merge` already knew the rule: the engine label was guarded with
`!existing.engines.contains(&engine)`. Put the score behind the same
guard, so each engine contributes its best rank once. Cross-engine
merging and the longest-snippet rule are untouched.
@leookun
leookun merged commit d23c341 into leookun:main Sep 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MaxFreedomPollard @leookun