EC: cache the full GET_SHARED_FILES / GET_DLOAD_QUEUE responses for FULL fetches - #731
Closed
got3nks wants to merge 9 commits into
Closed
EC: cache the full GET_SHARED_FILES / GET_DLOAD_QUEUE responses for FULL fetches#731got3nks wants to merge 9 commits into
got3nks wants to merge 9 commits into
Conversation
Adds the data + plumbing for skipping unchanged files in INC_UPDATE responses (amule-project#713). No callers yet, no behavior change — pure infrastructure to make the next two commits reviewable on their own. Each CKnownFile carries an atomic uint64 generation counter. A process- wide atomic global counter ensures every Mark…() yields a strictly- ascending value across all files and all threads (CFileStatistic counters tick from upload-disk-IO + hashing threads, not just main). `MarkECChanged()` uses a single fetch_add(1) + store rather than a load / increment / store so concurrent mutations from different threads can't collide on the same gen. Memory cost: 8 bytes per CKnownFile (~700 KB for a 91 k-file shareset). One additional atomic uint64 at file scope. Refs amule-project#713.
Wires `MarkECChanged()` into each storage primitive that writes a field
exported via `CEC_SharedFile_Tag` / `CEC_PartFile_Tag`. Still no behavior
change on the EC dispatch side — the next commit consumes the flag.
Sites covered:
* `CFileStatistic::AddRequest / AddAccepted / AddTransferred`
(EC_TAG_KNOWNFILE_REQ_COUNT, _ACCEPT_COUNT, _XFERRED and their _ALL
counterparts). `AddTransferred` is the hottest of the three —
every uploaded chunk flips the flag. The Mark op is a single
atomic fetch_add+store so the cost is small even on multi-slot
upload nodes.
* `CKnownFile::AddUploadingClient / RemoveUploadingClient`
(EC_TAG_KNOWNFILE_ON_QUEUE = m_ClientUploadList.size()).
* `CKnownFile::SetFileName` (EC_TAG_PARTFILE_NAME and ed2k:// link).
* `CKnownFile::SetFilePath` (EC_TAG_KNOWNFILE_FILENAME on non-partfiles).
* `CKnownFile::SetUpPriority / SetAutoUpPriority` (EC_TAG_KNOWNFILE_PRIO,
which folds the auto flag in as +10).
* `CKnownFile::SetFileCommentRating` (EC_TAG_KNOWNFILE_COMMENT / _RATING).
* `CKnownFile::UpdatePartsInfo` (EC_TAG_KNOWNFILE_COMPLETE_SOURCES{,LOW,HIGH}).
* `m_pAICHHashSet->SetMasterHash` call sites in KnownFile.cpp /
PartFile.cpp and `SetAICHHashset` swap in amule.cpp's hash-task
completion (EC_TAG_KNOWNFILE_AICH_MASTERHASH).
* `CPartFile::Process` — coarse hook at the per-second per-partfile
tick. Partfiles have ~20 EC-exported fields that change
independently on active downloads; the per-second tick is the
right granularity and avoids hooking each of them.
A 60 s periodic backstop in commit (3/3) catches anything missed here.
Refs amule-project#713.
Consumes the per-file `m_ecGen` from commit (1/3) and the `MarkECChanged`
hooks from commit (2/3): `Get_EC_Response_GetUpdate` now skips files
whose generation is at or below the highest gen this connection has
already reflected in its `CObjTagMap`. For a 91 k-file shareset with a
few hundred actively-transferring files at any moment, the per-cycle
iteration goes from O(N) to O(files-that-actually-changed) — typically
a 99 %+ reduction.
Mechanism per connection:
* snapshot = CKnownFile::s_globalEcGen at request start
* threshold = m_lastEcGenSeen carried from the previous response
* iterate only files where file.m_ecGen > threshold
* m_lastEcGenSeen := snapshot after iteration
* files whose m_ecGen changes mid-iteration land above the snapshot
and get picked up on the next request
Backstop: every kFullEcSyncIntervalSeconds (60 s) every connection
runs an unconditional full sweep regardless of gen. Worst-case
staleness if a `MarkECChanged()` hook was missed somewhere is bounded
to that interval rather than indefinite. The interval is short enough
to keep amulegui's UI from drifting noticeably and long enough that
the 99 % skip win still dominates the per-cycle workload.
Wire format unchanged; INC_UPDATE responses now contain only the
files that changed since the last response, which is exactly what the
delta protocol was designed to convey but the implementation used to
emit empty tag bodies for unchanged files anyway. Backwards-compatible
with any client version (the client side already correctly handles
missing-file-tag = "no change").
Refs amule-project#713.
…t-in
Commit 3 broke amulegui on big libraries because amulegui infers
deletions from absence: any file ID missing from an INC_UPDATE response
is removed from the local view. Skipping unchanged files entirely meant
most of the shareset disappeared every cycle (then the 5-min backstop
re-added them), wedging the GUI's list-widget at 100% CPU in
std::unordered_map churn.
Two-pronged fix preserving the optimization in both compat and opt-in
paths:
* Backward compat (old amulegui / amuleweb / third-party client):
unchanged files get a 5-byte "alive marker" tag — EC_TAG_KNOWNFILE /
EC_TAG_PARTFILE with the ECID and no children. The legacy client
already gates updates on `if (tag->HasChildTags())`, so empty tags
count as "present" without triggering the per-item update. Keeps the
expensive CEC_SharedFile_Tag construction + encoder pass off the
hot path (the actual perf win); only pays a tiny per-file wire cost.
* Partial-update protocol opt-in:
- New EC_TAG_CAN_PARTIAL_UPDATE capability advertised by the client
at auth, mirrored back by the server in AUTH_OK if it understands
the protocol.
- When negotiated, server skips unchanged files entirely and emits
explicit EC_TAG_FILE_REMOVED markers (top-level, value = ECID)
for files that disappeared since the previous cycle.
CECServerSocket tracks the previous cycle's snapshot in
m_lastSentFileIds.
- amulegui clients in this mode skip the bulk "missing == deleted"
loop and process explicit removals only.
Also drops the 5-minute paranoia backstop. With explicit removal
markers and alive markers covering both protocol paths, the backstop's
only remaining job was hiding missed MarkECChanged() hooks — making
those bugs harder to find rather than easier.
Negotiation matrix:
new server + new client → partial-update (full win, ~0 work
per-cycle for static files)
new server + old client → alive markers (almost full win, cheap
per-file wire cost)
old server + new client → server ignores capability, doesn't mirror
it back; client falls back to bulk-deletion path
old server + old client → unchanged
amuleweb drives its INC_UPDATE polling through `EC_OP_GET_SHARED_FILES` + `EC_OP_GET_DLOAD_QUEUE` with `EC_DETAIL_UPDATE` rather than the `EC_OP_GET_UPDATE` path amulegui uses. Commit (4/5) wired the partial- update protocol only into `Get_EC_Response_GetUpdate`, so amuleweb kept iterating all files every cycle even on huge libraries. Extend the same skip-unchanged + `EC_TAG_FILE_REMOVED` logic into `Get_EC_Response_GetSharedFiles` and `Get_EC_Response_GetDownloadQueue`, guarded on the same `m_partialUpdateActive` flag and only kicking in for `EC_DETAIL_UPDATE` requests (so amulecmd's `show shared` and other `EC_DETAIL_FULL` callers are untouched). Per-path state because the three handlers fire on independent cadences: * `m_lastEcGenSeen` + `m_lastSentFileIds` — amulegui's `EC_OP_GET_UPDATE` * `m_lastEcGenSeenShared` + `m_lastSentSharedFileIds` — amuleweb's `EC_OP_GET_SHARED_FILES` * `m_lastEcGenSeenPart` + `m_lastSentPartFileIds` — amuleweb's `EC_OP_GET_DLOAD_QUEUE` amuleweb's `UpdatableItemsContainer::ProcessUpdate` learns the same two-path behaviour: when the negotiated capability is active, drain `EC_TAG_FILE_REMOVED` markers into a buffered removal set and run a single sweep over the live list only when removals exist; otherwise the legacy "missing == deleted" loop stays correct (server still emits diff-empty alive-marker tags via the existing encoder path). `CaMuleExternalConnector::IsServerPartialUpdateActive()` exposes `CRemoteConnect::ServerSupportsPartialUpdate()` to the container. Compat matrix unchanged from (4/5): the negotiation is symmetric, so old amuleweb + new amuled and new amuleweb + old amuled both fall back to the legacy path automatically.
…ction setters
Reverse audit of every EC_DETAIL_UPDATE field surfaced nine mutation
sites that change exported state without marking the partfile changed.
For an active partfile the per-tick `MarkECChanged()` at the top of
`CPartFile::Process()` covered them implicitly, but `DownloadQueue::
Process()` gates `Process()` to `PS_READY`/`PS_EMPTY` only — paused,
stopped, hashing, and completed partfiles never auto-mark, so a user
toggling pause/resume/priority/category from amulegui or amuleweb
would stay invisible until something else on the file changed.
The 5-minute paranoia backstop (removed in 4/5) used to paper over
this; explicit hooks are the correct fix.
Sites covered:
* `CPartFile::StopFile` — EC_TAG_PARTFILE_STOPPED / _STATUS
* `CPartFile::PauseFile` — EC_TAG_PARTFILE_STOPPED / _STATUS
* `CPartFile::ResumeFile` — EC_TAG_PARTFILE_STOPPED / _STATUS
* `CPartFile::SetDownPriority` — EC_TAG_PARTFILE_PRIO
* `CPartFile::SetAutoDownPriority` — EC_TAG_PARTFILE_PRIO (auto flag
folded in via +10 offset)
* `CPartFile::SetA4AFAuto` — EC_TAG_PARTFILE_A4AFAUTO
* `CPartFile::SetCategory` — EC_TAG_PARTFILE_CAT
* `CPartFile::SetHashingProgress` — EC_TAG_PARTFILE_HASHED_PART_COUNT
(runs during hashing, outside
Process()'s gate)
* `CPartFile::UpdateFileRatingCommentAvail` — EC_TAG_PARTFILE_COMMENTS
(fires when sources deliver
ratings/comments via OP_MESSAGE)
All hooks are conditional on actual value change to keep the
generation counter from incrementing on no-op writes.
…ULL fetches amulecmd's `show shared` and `show DL` (and any other `EC_OP_GET_SHARED_FILES` / `EC_OP_GET_DLOAD_QUEUE` request issued at `EC_DETAIL_FULL` with no `EC_TAG_KNOWNFILE` / `EC_TAG_PARTFILE` queryitems) iterate the full file list and rebuild every per-file tag tree from scratch on every invocation. On a 91 k-file shareset that's the ~10 s of CPU per amulecmd `show shared` invocation profiled in issue amule-project#713. The work is fully redundant when the shareset hasn't changed since the previous request — which is the common case for amulecmd, since each invocation is a fresh short-lived EC connection with no client-side state to incrementalize against. This commit adds a daemon-wide cache keyed on `CKnownFile:: GetGlobalECGen()` (the process-wide change-generation counter introduced by PR amule-project#727): * One CECFullResponseCache instance per opcode (shared files, download queue). Each wraps a builder function that constructs a self-contained CECPacket from the current daemon state. * `Get()` returns a shared_ptr<const CECPacket>. If `s_globalEcGen` matches the cached generation, returns the cached packet immediately; otherwise rebuilds outside the lock and stores under a short critical section. * Both EC_OP_GET_SHARED_FILES and EC_OP_GET_DLOAD_QUEUE handlers check the fast path first: `detail == EC_DETAIL_FULL && queryitems is empty` -> SendPacket the cached tree and return NULL from ProcessRequest2 (skipping the framework's double-send). Anything else (EC_DETAIL_UPDATE, EC_DETAIL_INC_UPDATE, or any queryitems populated by amuleweb's phase-3 follow-up) stays on the existing per-connection path. Expected impact: * `show shared` on a 91 k library: ~10 s -> <100 ms in steady state. First request after any per-file change pays the rebuild cost (~10 s once), subsequent requests are cached again. * `show DL`: cache hit on idle / paused queues and back-to-back invocations. With active downloads CPartFile::Process() ticks `s_globalEcGen` ~1 Hz so the cache invalidates frequently, but the overhead is still bounded by at-most-one rebuild per cycle. Bypassed paths: * EC_DETAIL_UPDATE: amuleweb / amulegui INC_UPDATE. Per-connection diffs by definition; not cacheable cross-connection. * Requests with queryitems populated: amuleweb phase-3 follow-up for newly discovered IDs, asking for a small filtered subset. Memory cost: ~30 MB resident for the shared-files cache on a 91 k library; a few KB for the download-queue cache. Single copy daemon- wide, refcounted; freed automatically when superseded by a rebuild. Depends on amule-project#727 for `CKnownFile::GetGlobalECGen()` and the MarkECChanged hooks that drive its invalidation. Refs amule-project#713.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
May 27, 2026
…EUE paths amulecmd's `show shared` and `show DL` (and any other `EC_OP_GET_SHARED_FILES` / `EC_OP_GET_DLOAD_QUEUE` request issued at `EC_DETAIL_FULL` with no `EC_TAG_KNOWNFILE` / `EC_TAG_PARTFILE` queryitems) iterate every file and rebuild every per-file tag tree from scratch on every invocation. On a 91k-file shareset that's the ~10 s of CPU per amulecmd `show shared` invocation profiled in issue amule-project#713 — and the work is fully wasted when the shareset hasn't changed since the previous request. This adds a daemon-wide per-file bytes cache keyed off `CKnownFile::s_globalEcGen` (the change-generation counter from the INC_UPDATE work). Per file the cache holds one pre-serialized wire- format blob, freshness-stamped with the file's `m_ecGen` at build time. On each request: * Snapshot the current shareset / download queue. * For each file: reuse the cached blob if its gen still matches; otherwise build a fresh CEC_SharedFile_Tag / CEC_PartFile_Tag, serialize it via the new CECMemSocket, and store. Only the entries whose file gen advanced since the last request are rebuilt (the same per-file freshness primitive INC_UPDATE uses). * Concatenate the per-file blobs and feed them through the connection's socket via SendCachedBodyResponse, which mirrors WritePacket's flag-byte / length-header / per-connection deflate dance from WriteBuffer outward — so the per-connection ZLIB- bypass on local peers still applies to the concatenated stream. Bypassed paths (fall through to the existing live builders): * EC_DETAIL_UPDATE — amuleweb / amulegui INC_UPDATE. Per-connection diff state by definition, not shareable. * Requests with queryitems populated — amuleweb's phase-3 follow-up for newly discovered IDs, asking for a small filtered subset. * Connections that didn't negotiate EC_FLAG_UTF8_NUMBERS + EC_FLAG_LARGE_TAG_COUNT (the wire format the cached blobs assume). Every modern client advertises both at auth; very old clients drop to the live path automatically. New EC-library pieces: * CECMemSocket — a CECSocket subclass that captures all I/O into an in-memory vector. Used at build time to serialize one tag into a self-contained byte blob. * CECSocket::SendCachedBodyResponse — public helper that emits a precomputed body (opcode + child-count + concatenated blobs) through the existing socket buffer / compression machinery. * CECTag::Serialize — public wrapper around the protected WriteTag primitive (was already the right shape for serializing a single self-contained tag). * CECSocket::SetTxFlags — protected setter so daemon-internal subclasses can pre-set the per-packet wire format before invoking WriteBuffer / WriteNumber directly (without going through WritePacket). * `friend class CECMemSocket` on CECSocket so the mem sink can call FlushBuffers / OnOutput to drain the in-flight buffer chain into its capture vector. Memory cost: ~30 MB resident on a 91k library for the shared-files cache; a few KB for the download queue. Single copy daemon-wide, shared_ptr-refcounted; orphan entries pruned after each request via CECFullResponseCache::PruneOutsideOf. Cache invalidation rate: tracks s_globalEcGen, which bumps on every per-file change including chunk transfers. For idle / mostly-paused daemons the cache holds; busy seeders rebuild many per-file entries per second but at most one rebuild per file per change. The per-file granularity (vs the earlier whole-blob cache attempt in amule-project#731) means concurrent activity on one file doesn't invalidate the cached blobs for the other 90,999 files in the shareset. Local smoke test (3-file shareset): * `show shared` first call: 0.446 s (cold cache build). * `show shared` subsequent: 0.079 s (cache hit). * Add file, reload, `show shared`: new file present, cache adapts. * Delete file, reload, `show shared`: file gone, prune drops it. * amulegui INC_UPDATE path stays at <5 % CPU (cache bypassed correctly). Refs amule-project#713.
6 tasks
mrjimenez
pushed a commit
that referenced
this pull request
May 27, 2026
…EUE paths amulecmd's `show shared` and `show DL` (and any other `EC_OP_GET_SHARED_FILES` / `EC_OP_GET_DLOAD_QUEUE` request issued at `EC_DETAIL_FULL` with no `EC_TAG_KNOWNFILE` / `EC_TAG_PARTFILE` queryitems) iterate every file and rebuild every per-file tag tree from scratch on every invocation. On a 91k-file shareset that's the ~10 s of CPU per amulecmd `show shared` invocation profiled in issue #713 — and the work is fully wasted when the shareset hasn't changed since the previous request. This adds a daemon-wide per-file bytes cache keyed off `CKnownFile::s_globalEcGen` (the change-generation counter from the INC_UPDATE work). Per file the cache holds one pre-serialized wire- format blob, freshness-stamped with the file's `m_ecGen` at build time. On each request: * Snapshot the current shareset / download queue. * For each file: reuse the cached blob if its gen still matches; otherwise build a fresh CEC_SharedFile_Tag / CEC_PartFile_Tag, serialize it via the new CECMemSocket, and store. Only the entries whose file gen advanced since the last request are rebuilt (the same per-file freshness primitive INC_UPDATE uses). * Concatenate the per-file blobs and feed them through the connection's socket via SendCachedBodyResponse, which mirrors WritePacket's flag-byte / length-header / per-connection deflate dance from WriteBuffer outward — so the per-connection ZLIB- bypass on local peers still applies to the concatenated stream. Bypassed paths (fall through to the existing live builders): * EC_DETAIL_UPDATE — amuleweb / amulegui INC_UPDATE. Per-connection diff state by definition, not shareable. * Requests with queryitems populated — amuleweb's phase-3 follow-up for newly discovered IDs, asking for a small filtered subset. * Connections that didn't negotiate EC_FLAG_UTF8_NUMBERS + EC_FLAG_LARGE_TAG_COUNT (the wire format the cached blobs assume). Every modern client advertises both at auth; very old clients drop to the live path automatically. New EC-library pieces: * CECMemSocket — a CECSocket subclass that captures all I/O into an in-memory vector. Used at build time to serialize one tag into a self-contained byte blob. * CECSocket::SendCachedBodyResponse — public helper that emits a precomputed body (opcode + child-count + concatenated blobs) through the existing socket buffer / compression machinery. * CECTag::Serialize — public wrapper around the protected WriteTag primitive (was already the right shape for serializing a single self-contained tag). * CECSocket::SetTxFlags — protected setter so daemon-internal subclasses can pre-set the per-packet wire format before invoking WriteBuffer / WriteNumber directly (without going through WritePacket). * `friend class CECMemSocket` on CECSocket so the mem sink can call FlushBuffers / OnOutput to drain the in-flight buffer chain into its capture vector. Memory cost: ~30 MB resident on a 91k library for the shared-files cache; a few KB for the download queue. Single copy daemon-wide, shared_ptr-refcounted; orphan entries pruned after each request via CECFullResponseCache::PruneOutsideOf. Cache invalidation rate: tracks s_globalEcGen, which bumps on every per-file change including chunk transfers. For idle / mostly-paused daemons the cache holds; busy seeders rebuild many per-file entries per second but at most one rebuild per file per change. The per-file granularity (vs the earlier whole-blob cache attempt in #731) means concurrent activity on one file doesn't invalidate the cached blobs for the other 90,999 files in the shareset. Local smoke test (3-file shareset): * `show shared` first call: 0.446 s (cold cache build). * `show shared` subsequent: 0.079 s (cache hit). * Add file, reload, `show shared`: new file present, cache adapts. * Delete file, reload, `show shared`: file gone, prune drops it. * amulegui INC_UPDATE path stays at <5 % CPU (cache bypassed correctly). Refs #713.
ngosang
added a commit
to ngosang/amule
that referenced
this pull request
Jul 31, 2026
…e-project#731) The download detail's Comments tab re-fetched GET /downloads/{hash}/comments on every tick of the downloads store, so an open tab on an active download issued roughly one request per second. amuled already emits a comments_updated SSE event whose payload is byte-for-byte that same body (EVENTS.md), and it was simply not wired up on the client. - events.js: subscribe to comments_updated and republish it on the "comments:updated" store key, matching the existing search:result / log:appended one-off event pattern. - download-detail.js: DownloadComments now fetches once per hash and applies the event payload directly when its hash matches, so retrieved Kad notes and source-reported comments both land without polling. Because EqualComments is deliberately excluded from EqualDownload, notes arriving mid-search on an otherwise idle (paused/stopped) download previously stayed invisible until the lookup finished; they now show up as they arrive. - The "Searching Kad…" button state reads kad_comment_search_running from the downloads store rather than from the comments body: the Comments tab pins liveTick to 0, so the detail object never refreshes while that tab is open, whereas the list item does (the flag is part of EqualDownload, so its start -> finish edge arrives as download_updated). Verified against a running amuled: one GET /comments over a 90s open tab (previously one per tick), the button flipping to "Searching Kad…" and back on a real Kad lookup, and no console errors.
got3nks
pushed a commit
to got3nks/amule
that referenced
this pull request
Jul 31, 2026
…r.cpp (amule-project#675) (amule-project#725) * chore(gui): delete unreachable bitmap functions/entries from muuli_wdr.cpp First slice of the icon-system cleanup scoped in amule-project#675: remove code with zero call sites anywhere in the tree, before any wxArtProvider migration work starts. - muleToolbar(): whole function unused -- superseded by the main wxToolBar setup in amuleDlg.cpp; nothing calls it. - moreImages(): whole function unused, both of its two icon entries. - amuleDlgImages(): 21 of 35 index blocks have no caller anywhere (0-13, 16, 17, 19, 27, 28, 31, 34). The 14 live ones are untouched -- 10 of those (20-26, 29, 32, 33) are already the fallback path inside amuleDlg.cpp's Add_Skin_Icon, which prefers a wxArtProvider/SVG lookup first; the other 4 (14, 15, 18, 30) are still called directly. - amuleSpecial(): 6 of 26 index blocks have no caller (6, 7, 8, 9, 18, 20) -- checked both direct call sites and the PrefsUnifiedDlg.cpp fallback table (pages[].m_imageidx), which uses neither. convert_xpm in PartFileConvertDlg.cpp was on the same "no literal grep hits" list initially but is not actually dead -- SetIcon(wxICON (convert)) reaches it via the wxICON macro's token-pasting (X##_xpm), invisible to a plain identifier search. Caught by a full build failing on the undeclared identifier, not by inspection; left untouched. Deletes 1373 lines (~16% of the file). No behavior change: every touched entry was unreachable code. clang-format v18 clean; full amule build verified (macOS, CLIENT_GUI unaffected since neither touched symbol is CLIENT_GUI-only). * chore(po): regenerate catalogs after muuli_wdr.cpp dead-code removal muleToolbar() duplicated several msgid source-location references (Networks, Searches, Downloads Window, etc.) already present via amuleDlg.cpp's own toolbar setup. Deleting it drops those now-stale #: comments; no msgid added, removed, or retranslated -- verified via unchanged msgid count (1903 before/after) and a diff limited to source-location comments and POT-Creation-Date. Regenerated after rebasing onto upstream/master to pick up po/ changes from amule-project#723/amule-project#724/amule-project#726/amule-project#728/amule-project#730/amule-project#731, which had drifted our prior regeneration out of sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
amulecmd's
show sharedandshow DL(and any otherEC_OP_GET_SHARED_FILES/EC_OP_GET_DLOAD_QUEUErequest issued atEC_DETAIL_FULLwith noEC_TAG_KNOWNFILE/EC_TAG_PARTFILEqueryitems) iterate the full file list and rebuild every per-file tag tree from scratch on every invocation. On a 91k-file shareset that's the ~10 s of CPU per amulecmdshow sharedinvocation profiled in #713.The work is fully redundant when the shareset hasn't changed since the previous request — which is the common case for amulecmd, since each invocation is a fresh short-lived EC connection with no client-side state to incrementalize against.
This PR adds a daemon-wide cache keyed on
CKnownFile::GetGlobalECGen()(the process-wide change-generation counter introduced by #727):CECFullResponseCacheinstance per opcode (shared files, download queue). Each wraps a builder function that constructs a self-containedCECPacketfrom the current daemon state.Get()returns ashared_ptr<const CECPacket>. Ifs_globalEcGenmatches the cached generation, returns the cached packet immediately; otherwise rebuilds outside the lock and stores under a short critical section.EC_OP_GET_SHARED_FILESandEC_OP_GET_DLOAD_QUEUEhandlers check the fast path first:detail == EC_DETAIL_FULL && queryitems is empty→SendPacketthe cached tree and returnNULLfromProcessRequest2(skipping the framework's double-send). Anything else (EC_DETAIL_UPDATE,EC_DETAIL_INC_UPDATE, or any queryitems populated by amuleweb's phase-3 follow-up) stays on the existing per-connection path.Expected impact
show sharedon a 91k library: ~10 s → <100 ms in steady state. First request after any per-file change pays the rebuild cost (~10 s once), subsequent requests are cached again.show DL: cache hit on idle / paused queues and back-to-back invocations. With active downloadsCPartFile::Process()tickss_globalEcGen~1 Hz so the cache invalidates frequently, but the overhead is still bounded by at-most-one rebuild per cycle.Bypassed paths
EC_DETAIL_UPDATE— amuleweb / amulegui INC_UPDATE, per-connection diffs by definition.Memory cost
Single copy daemon-wide, refcounted via
shared_ptr; freed automatically when superseded by a rebuild.Dependencies
Depends on #727 for
CKnownFile::GetGlobalECGen()and theMarkECChanged()hooks that drive its invalidation. Built on top offeat/ec-skip-unchanged-files.Test plan
amuled,amulegui,amulecmd,amuleweb— clean compile.amulecmd show sharedfirst call builds the cache (~0.43 s), second call hits (~0.085 s). Same forshow DL.reload shared, nextshow sharedrebuilds correctly (4 files vs prior 3). Subsequent calls hit the new cache.Partial update: yes, INC responses 100-1400 bytes; the cache code is bypassed).time amulecmd -c "show shared"should drop from ~10 s to <1 s in steady state.Refs #713, depends on #727.