perf(ec): skip unchanged search results on the multi-search union poll - #723

An idle amuleGUI sitting on the Search panel spends roughly 14 KB per poll for every 1000 search results it holds, purely to tell the daemon that nothing has changed. At the 3 s cadence that is about 17 MB/hour per 1000 results, per connected client — and it scales with how much the user has open, not with how much changed.

Every figure below was measured with amule-ec-profiler, a pass-through EC proxy that sizes each packet and breaks it down per tag, so the costs here are observed on the wire rather than reasoned from the source.

What it costs today

Measured against a daemon holding two finished searches and ~900 results, with the search tab open and nothing running:

Nothing else — every mutable field diffs away correctly through the per-connection valuemap. What remains is a fixed ~14 bytes per result, per poll, forever, for results nobody is looking at.

Why both halves were there

The envelope was load-bearing. CRemoteContainer::ProcessUpdate deletes any item absent from the reply, so omitting an unchanged result made the client drop it from the tab. Absence was the deletion protocol.

EC_TAG_SEARCH_ID was then emitted unconditionally, outside the valuemap, so a result re-created after such a deletion could still be attributed to a tab. It was guarding against a case the first half caused.

Both fall away once removal is explicit.

The change

The union now skips a result whose valuemap diff is empty and that the client already has, emits EC_TAG_FILE_REMOVED for results the core no longer holds, and sends EC_TAG_SEARCH_ID only on a result's first appearance. CSearchListRem::ProcessUpdate deletes only on those tombstones instead of sweeping by absence. A tombstone also clears that ECID's valuemap, so a re-created result cannot inherit a diff that would strip the fields it needs — the same hazard freshEcids guards on the file path.

This reuses the existing knownfile tombstone rather than inventing a second one: the meaning ("this ECID is gone") and the payload are identical.

Compatibility

Negotiated as its own capability, EC_TAG_CAN_PARTIAL_SEARCH, not by reusing EC_TAG_CAN_PARTIAL_UPDATE. That existing flag is advertised unconditionally by every RemoteConnect client and means the file INC_UPDATE stream. An already-released amuleGUI advertises it, speaks multi-search, and still deletes any result missing from the reply — so reusing it would make such a client silently lose its search results against a newer daemon.

amuleweb never sets EC_TAG_CAN_MULTI_SEARCH, so it stays on the per-search EC_DETAIL_UPDATE branch; amuleapi addresses searches by id at EC_DETAIL_FULL.

Measured

Same daemon, same searches, before and after:

The two runs held different amounts, which is the point: master's cost is per result, so it has to be compared per result. This branch's 10 bytes is a bare envelope with a zero tag count — a constant, unchanged whether the client holds 800 results or 1500. The cost is now proportional to what changed, not to what is open.

Mechanism confirmed on the wire rather than inferred: closing two tabs produced 1223 EC_TAG_FILE_REMOVED tombstones once each; a subsequent search delivered 313 EC_TAG_SEARCHFILE against 269 EC_TAG_SEARCH_ID, the 44-tag gap being updates to already-known results correctly omitting the id.

Testing

• amuleGUI against a live daemon on ed2k + Kad: four searches started, two tabs closed, a fourth search run. Results stay put, close removes them, new results land in the right tab.
• The same daemon with an amuleGUI built from 70c1ec09a (this branch's base, so the only difference is this change): 11,616 B per poll of envelope + id, zero tombstones, results intact. The fallback is per connection, not global.
• amuleapi curl suite 32/32 phases, 19-search.sh 108/108.
• clang-format v18 and Tier-2 clang-tidy on changed lines, both clean.

Two things stated rather than measured, so they can be challenged cheaply:

• Old daemon, new client is argued, not run. m_serverPartialSearch has exactly two writes — false in the constructor and true only when the daemon echoes the capability in AUTH_OK — so against an old daemon the client falls through to the unmodified base ProcessUpdate. The premise that an old daemon ignores an unknown AUTH_REQ tag is the same mechanism ten existing capability tags already rely on.
• Kad comments are unaffected by construction: that block bypasses the valuemap, so whenever there is anything to report the tag has children and the skip cannot fire. The one case it does not cover — a Kad comment search that finishes finding nothing never telling the client it ended — behaves identically on master, since the client only acts on a tag that is present.

Results carrying Kad comments are still re-sent in full every poll, because that block has no valuemap. That is unchanged here, but it will now be the only remaining per-poll cost and is worth a separate look.