EC: per-file bytes cache for the FULL GET_SHARED_FILES / GET_DLOAD_QUEUE paths - #736
Merged
mrjimenez merged 1 commit intoMay 27, 2026
Conversation
…EUE paths amulecmd's `show shared` and `show DL` (and any other `EC_OP_GET_SHARED_FILES` / `EC_OP_GET_DLOAD_QUEUE` request issued at `EC_DETAIL_FULL` with no `EC_TAG_KNOWNFILE` / `EC_TAG_PARTFILE` queryitems) iterate every file and rebuild every per-file tag tree from scratch on every invocation. On a 91k-file shareset that's the ~10 s of CPU per amulecmd `show shared` invocation profiled in issue amule-project#713 — and the work is fully wasted when the shareset hasn't changed since the previous request. This adds a daemon-wide per-file bytes cache keyed off `CKnownFile::s_globalEcGen` (the change-generation counter from the INC_UPDATE work). Per file the cache holds one pre-serialized wire- format blob, freshness-stamped with the file's `m_ecGen` at build time. On each request: * Snapshot the current shareset / download queue. * For each file: reuse the cached blob if its gen still matches; otherwise build a fresh CEC_SharedFile_Tag / CEC_PartFile_Tag, serialize it via the new CECMemSocket, and store. Only the entries whose file gen advanced since the last request are rebuilt (the same per-file freshness primitive INC_UPDATE uses). * Concatenate the per-file blobs and feed them through the connection's socket via SendCachedBodyResponse, which mirrors WritePacket's flag-byte / length-header / per-connection deflate dance from WriteBuffer outward — so the per-connection ZLIB- bypass on local peers still applies to the concatenated stream. Bypassed paths (fall through to the existing live builders): * EC_DETAIL_UPDATE — amuleweb / amulegui INC_UPDATE. Per-connection diff state by definition, not shareable. * Requests with queryitems populated — amuleweb's phase-3 follow-up for newly discovered IDs, asking for a small filtered subset. * Connections that didn't negotiate EC_FLAG_UTF8_NUMBERS + EC_FLAG_LARGE_TAG_COUNT (the wire format the cached blobs assume). Every modern client advertises both at auth; very old clients drop to the live path automatically. New EC-library pieces: * CECMemSocket — a CECSocket subclass that captures all I/O into an in-memory vector. Used at build time to serialize one tag into a self-contained byte blob. * CECSocket::SendCachedBodyResponse — public helper that emits a precomputed body (opcode + child-count + concatenated blobs) through the existing socket buffer / compression machinery. * CECTag::Serialize — public wrapper around the protected WriteTag primitive (was already the right shape for serializing a single self-contained tag). * CECSocket::SetTxFlags — protected setter so daemon-internal subclasses can pre-set the per-packet wire format before invoking WriteBuffer / WriteNumber directly (without going through WritePacket). * `friend class CECMemSocket` on CECSocket so the mem sink can call FlushBuffers / OnOutput to drain the in-flight buffer chain into its capture vector. Memory cost: ~30 MB resident on a 91k library for the shared-files cache; a few KB for the download queue. Single copy daemon-wide, shared_ptr-refcounted; orphan entries pruned after each request via CECFullResponseCache::PruneOutsideOf. Cache invalidation rate: tracks s_globalEcGen, which bumps on every per-file change including chunk transfers. For idle / mostly-paused daemons the cache holds; busy seeders rebuild many per-file entries per second but at most one rebuild per file per change. The per-file granularity (vs the earlier whole-blob cache attempt in amule-project#731) means concurrent activity on one file doesn't invalidate the cached blobs for the other 90,999 files in the shareset. Local smoke test (3-file shareset): * `show shared` first call: 0.446 s (cold cache build). * `show shared` subsequent: 0.079 s (cache hit). * Add file, reload, `show shared`: new file present, cache adapts. * Delete file, reload, `show shared`: file gone, prune drops it. * amulegui INC_UPDATE path stays at <5 % CPU (cache bypassed correctly). Refs amule-project#713.
got3nks
marked this pull request as ready for review
May 27, 2026 11:53
3 tasks
1 task
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Jun 4, 2026
…ndex Adds 55+ merged PRs to the 3.0.0 changelog since the last update (amule-project#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796 (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths). - Bug Fixes & Stability: post-amule-project#744 fixes including EC notification leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728), amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760), PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788, amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756), Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833, amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774). - Translations: late-cycle wave covering French/Turkish manpages (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for date+version drift (amule-project#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Jun 4, 2026
…ndex Adds 55+ merged PRs to the 3.0.0 changelog since the last update (amule-project#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796 (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths). - Bug Fixes & Stability: post-amule-project#744 fixes including EC notification leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728), amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760), PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788, amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756), Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833, amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774). - Translations: late-cycle wave covering French/Turkish manpages (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for date+version drift (amule-project#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
mrjimenez
pushed a commit
that referenced
this pull request
Jun 4, 2026
Adds 55+ merged PRs to the 3.0.0 changelog since the last update (#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for #785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), #794 (.dmg amuleweb path), #789 (<OS>-<arch> artifact naming), #780 / #796 (Windows DPI + comctl32 manifest), #784 (FHS share/amule paths). - Bug Fixes & Stability: post-#744 fixes including EC notification leak (#797), big-library scaling (#736, #840 superseding #728), amulegui ghost entries (#810, #819, #841, #824, #830, #760), PartFile early hash (#762), server protocol fixes (#835, #788, #721, #787), crypto stream UB (#779), UAF prevention (#756), Kad rotation (#795, #799/#805), GTK warning silencing (#833, #826/#836), and the clang-tidy worklist (#770, #772-#774). - Translations: late-cycle wave covering French/Turkish manpages (#753/#754/#776), Galician (#763), Slovenian (#771), pt-BR (#768/#775/#812), French (#811), plus man-page tooling for date+version drift (#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (#817/#818/#821/#828/#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with #746-#845 + #841.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Jul 31, 2026
The tree has no Java left. Its last .java file was the generated ECCodes.java, removed in e86646c along with the other committed EC headers, so the java-kotlin job has had nothing to scan since and fails on every run. Leaves actions, c-cpp and python, which is what the repository actually contains.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes the closed #731 — that PR was an early sketch that re-built the entire serialized response on any per-file change. This one is the per-file-bytes-cache design we ended up wanting: keyed off
CKnownFile::s_globalEcGenper-file, so concurrent activity on one file doesn't invalidate the cached blobs for the other 90,999.Summary
amulecmd's
show sharedandshow DL(and any otherEC_OP_GET_SHARED_FILES/EC_OP_GET_DLOAD_QUEUErequest issued atEC_DETAIL_FULLwith noEC_TAG_KNOWNFILE/EC_TAG_PARTFILEqueryitems) iterate every file and rebuild every per-file tag tree from scratch on every invocation. On a 91k-file shareset that's the ~10 s of CPU per amulecmdshow sharedinvocation profiled in #713 — and the work is fully wasted when the shareset hasn't changed since the previous request.This adds a daemon-wide per-file bytes cache keyed off
CKnownFile::s_globalEcGen(the change-generation counter from the INC_UPDATE work). Per file the cache holds one pre-serialized wire-format blob, freshness-stamped with the file'sm_ecGenat build time. On each request:CEC_SharedFile_Tag/CEC_PartFile_Tag, serialize it via the newCECMemSocket, and store. Only the entries whose file gen advanced since the last request are rebuilt (the same per-file freshness primitive the INC_UPDATE path uses).SendCachedBodyResponse, which mirrorsWritePacket's flag-byte / length-header / per-connection deflate dance fromWriteBufferoutward — so the per-connection ZLIB-bypass on local peers still applies to the concatenated stream.Bypassed paths (fall through to the existing live builders)
EC_DETAIL_UPDATE— amuleweb / amulegui INC_UPDATE. Per-connection diff state by definition, not shareable.EC_FLAG_UTF8_NUMBERS+EC_FLAG_LARGE_TAG_COUNT(the wire format the cached blobs assume). Every modern client advertises both at auth; very old clients drop to the live path automatically.New EC-library pieces
CECMemSocket— a CECSocket subclass that captures all I/O into an in-memory vector. Used at build time to serialize one tag into a self-contained byte blob.CECSocket::SendCachedBodyResponse— public helper that emits a precomputed body (opcode + child-count + concatenated blobs) through the existing socket buffer / compression machinery.CECTag::Serialize— public wrapper around the protectedWriteTagprimitive.CECSocket::SetTxFlags— protected setter so daemon-internal subclasses can pre-set the per-packet wire format before invokingWriteBuffer/WriteNumberdirectly.friend class CECMemSocketonCECSocketso the mem sink can callFlushBuffers/OnOutputto drain the in-flight buffer chain.Memory cost
~30 MB resident on a 91k library for the shared-files cache; a few KB for the download queue. Single copy daemon-wide,
shared_ptr-refcounted; orphan entries pruned after each request viaCECFullResponseCache::PruneOutsideOf.Cache invalidation rate
Tracks
s_globalEcGen, which bumps on every per-file change including chunk transfers. For idle / mostly-paused daemons the cache holds; busy seeders rebuild many per-file entries per second but at most one rebuild per file per change. The per-file granularity is the key win over the closed #731 whole-blob approach — activity on one file doesn't invalidate the other 90,999.Test plan
show shared0.446 s (cold cache build), second 0.079 s (cache hit). Same forshow DL.show shared; cache adapts.PruneOutsideOf.EC_DETAIL_UPDATEdoesn't enter the cache branch).time amulecmd -c "show shared"should drop from ~10 s to few secs in steady state, and individual per-file changes should invalidate only that file's blob (not the whole cache).Refs #713.