Skip to content

EC: per-file bytes cache for the FULL GET_SHARED_FILES / GET_DLOAD_QUEUE paths - #736

Merged
mrjimenez merged 1 commit into
amule-project:masterfrom
got3nks:feat/ec-full-response-cache-on-727
May 27, 2026
Merged

EC: per-file bytes cache for the FULL GET_SHARED_FILES / GET_DLOAD_QUEUE paths#736
mrjimenez merged 1 commit into
amule-project:masterfrom
got3nks:feat/ec-full-response-cache-on-727

Conversation

@got3nks

@got3nks got3nks commented May 27, 2026

Copy link
Copy Markdown
Contributor

Supersedes the closed #731 — that PR was an early sketch that re-built the entire serialized response on any per-file change. This one is the per-file-bytes-cache design we ended up wanting: keyed off CKnownFile::s_globalEcGen per-file, so concurrent activity on one file doesn't invalidate the cached blobs for the other 90,999.

Summary

amulecmd's show shared and show DL (and any other EC_OP_GET_SHARED_FILES / EC_OP_GET_DLOAD_QUEUE request issued at EC_DETAIL_FULL with no EC_TAG_KNOWNFILE / EC_TAG_PARTFILE queryitems) iterate every file and rebuild every per-file tag tree from scratch on every invocation. On a 91k-file shareset that's the ~10 s of CPU per amulecmd show shared invocation profiled in #713 — and the work is fully wasted when the shareset hasn't changed since the previous request.

This adds a daemon-wide per-file bytes cache keyed off CKnownFile::s_globalEcGen (the change-generation counter from the INC_UPDATE work). Per file the cache holds one pre-serialized wire-format blob, freshness-stamped with the file's m_ecGen at build time. On each request:

  • Snapshot the current shareset / download queue.
  • For each file: reuse the cached blob if its gen still matches; otherwise build a fresh CEC_SharedFile_Tag / CEC_PartFile_Tag, serialize it via the new CECMemSocket, and store. Only the entries whose file gen advanced since the last request are rebuilt (the same per-file freshness primitive the INC_UPDATE path uses).
  • Concatenate the per-file blobs and feed them through the connection's socket via SendCachedBodyResponse, which mirrors WritePacket's flag-byte / length-header / per-connection deflate dance from WriteBuffer outward — so the per-connection ZLIB-bypass on local peers still applies to the concatenated stream.

Bypassed paths (fall through to the existing live builders)

  • EC_DETAIL_UPDATE — amuleweb / amulegui INC_UPDATE. Per-connection diff state by definition, not shareable.
  • Requests with queryitems populated — amuleweb's phase-3 follow-up for newly discovered IDs, asking for a small filtered subset.
  • Connections that didn't negotiate EC_FLAG_UTF8_NUMBERS + EC_FLAG_LARGE_TAG_COUNT (the wire format the cached blobs assume). Every modern client advertises both at auth; very old clients drop to the live path automatically.

New EC-library pieces

  • CECMemSocket — a CECSocket subclass that captures all I/O into an in-memory vector. Used at build time to serialize one tag into a self-contained byte blob.
  • CECSocket::SendCachedBodyResponse — public helper that emits a precomputed body (opcode + child-count + concatenated blobs) through the existing socket buffer / compression machinery.
  • CECTag::Serialize — public wrapper around the protected WriteTag primitive.
  • CECSocket::SetTxFlags — protected setter so daemon-internal subclasses can pre-set the per-packet wire format before invoking WriteBuffer / WriteNumber directly.
  • friend class CECMemSocket on CECSocket so the mem sink can call FlushBuffers / OnOutput to drain the in-flight buffer chain.

Memory cost

~30 MB resident on a 91k library for the shared-files cache; a few KB for the download queue. Single copy daemon-wide, shared_ptr-refcounted; orphan entries pruned after each request via CECFullResponseCache::PruneOutsideOf.

Cache invalidation rate

Tracks s_globalEcGen, which bumps on every per-file change including chunk transfers. For idle / mostly-paused daemons the cache holds; busy seeders rebuild many per-file entries per second but at most one rebuild per file per change. The per-file granularity is the key win over the closed #731 whole-blob approach — activity on one file doesn't invalidate the other 90,999.

Test plan

  • macOS Debug build of amuled, amulegui, amulecmd, amuleweb — clean compile.
  • Loopback smoke test: small shareset, first show shared 0.446 s (cold cache build), second 0.079 s (cache hit). Same for show DL.
  • Add file + reload: new file appears in next show shared; cache adapts.
  • Delete file + reload: file gone; cache pruned via PruneOutsideOf.
  • amulegui INC_UPDATE path stays at <5 % CPU (cache bypassed correctly — EC_DETAIL_UPDATE doesn't enter the cache branch).
  • @Stoatwblr profile run on the 91k-file shareset: time amulecmd -c "show shared" should drop from ~10 s to few secs in steady state, and individual per-file changes should invalidate only that file's blob (not the whole cache).

Refs #713.

…EUE paths

amulecmd's `show shared` and `show DL` (and any other
`EC_OP_GET_SHARED_FILES` / `EC_OP_GET_DLOAD_QUEUE` request issued at
`EC_DETAIL_FULL` with no `EC_TAG_KNOWNFILE` / `EC_TAG_PARTFILE`
queryitems) iterate every file and rebuild every per-file tag tree
from scratch on every invocation. On a 91k-file shareset that's the
~10 s of CPU per amulecmd `show shared` invocation profiled in issue
amule-project#713 — and the work is fully wasted when the shareset hasn't changed
since the previous request.

This adds a daemon-wide per-file bytes cache keyed off
`CKnownFile::s_globalEcGen` (the change-generation counter from the
INC_UPDATE work). Per file the cache holds one pre-serialized wire-
format blob, freshness-stamped with the file's `m_ecGen` at build
time. On each request:

  * Snapshot the current shareset / download queue.
  * For each file: reuse the cached blob if its gen still matches;
    otherwise build a fresh CEC_SharedFile_Tag / CEC_PartFile_Tag,
    serialize it via the new CECMemSocket, and store. Only the
    entries whose file gen advanced since the last request are
    rebuilt (the same per-file freshness primitive INC_UPDATE uses).
  * Concatenate the per-file blobs and feed them through the
    connection's socket via SendCachedBodyResponse, which mirrors
    WritePacket's flag-byte / length-header / per-connection deflate
    dance from WriteBuffer outward — so the per-connection ZLIB-
    bypass on local peers still applies to the concatenated stream.

Bypassed paths (fall through to the existing live builders):

  * EC_DETAIL_UPDATE — amuleweb / amulegui INC_UPDATE. Per-connection
    diff state by definition, not shareable.
  * Requests with queryitems populated — amuleweb's phase-3 follow-up
    for newly discovered IDs, asking for a small filtered subset.
  * Connections that didn't negotiate EC_FLAG_UTF8_NUMBERS +
    EC_FLAG_LARGE_TAG_COUNT (the wire format the cached blobs
    assume). Every modern client advertises both at auth; very old
    clients drop to the live path automatically.

New EC-library pieces:

  * CECMemSocket — a CECSocket subclass that captures all I/O into
    an in-memory vector. Used at build time to serialize one tag
    into a self-contained byte blob.
  * CECSocket::SendCachedBodyResponse — public helper that emits a
    precomputed body (opcode + child-count + concatenated blobs)
    through the existing socket buffer / compression machinery.
  * CECTag::Serialize — public wrapper around the protected WriteTag
    primitive (was already the right shape for serializing a single
    self-contained tag).
  * CECSocket::SetTxFlags — protected setter so daemon-internal
    subclasses can pre-set the per-packet wire format before invoking
    WriteBuffer / WriteNumber directly (without going through
    WritePacket).
  * `friend class CECMemSocket` on CECSocket so the mem sink can call
    FlushBuffers / OnOutput to drain the in-flight buffer chain into
    its capture vector.

Memory cost: ~30 MB resident on a 91k library for the shared-files
cache; a few KB for the download queue. Single copy daemon-wide,
shared_ptr-refcounted; orphan entries pruned after each request via
CECFullResponseCache::PruneOutsideOf.

Cache invalidation rate: tracks s_globalEcGen, which bumps on every
per-file change including chunk transfers. For idle / mostly-paused
daemons the cache holds; busy seeders rebuild many per-file entries
per second but at most one rebuild per file per change. The per-file
granularity (vs the earlier whole-blob cache attempt in amule-project#731) means
concurrent activity on one file doesn't invalidate the cached blobs
for the other 90,999 files in the shareset.

Local smoke test (3-file shareset):

  * `show shared` first call: 0.446 s (cold cache build).
  * `show shared` subsequent: 0.079 s (cache hit).
  * Add file, reload, `show shared`: new file present, cache adapts.
  * Delete file, reload, `show shared`: file gone, prune drops it.
  * amulegui INC_UPDATE path stays at <5 % CPU (cache bypassed
    correctly).

Refs amule-project#713.
@got3nks
got3nks marked this pull request as ready for review May 27, 2026 11:53
@mrjimenez
mrjimenez merged commit 148b747 into amule-project:master May 27, 2026
7 checks passed
@got3nks
got3nks deleted the feat/ec-full-response-cache-on-727 branch May 27, 2026 15:15
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
…ndex

Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(amule-project#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg
  amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796
  (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths).

- Bug Fixes & Stability: post-amule-project#744 fixes including EC notification
  leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728),
  amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760),
  PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788,
  amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756),
  Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833,
  amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR
  (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for
  date+version drift (amule-project#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
…ndex

Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(amule-project#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg
  amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796
  (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths).

- Bug Fixes & Stability: post-amule-project#744 fixes including EC notification
  leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728),
  amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760),
  PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788,
  amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756),
  Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833,
  amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR
  (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for
  date+version drift (amule-project#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
mrjimenez pushed a commit that referenced this pull request Jun 4, 2026
Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for #785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), #794 (.dmg
  amuleweb path), #789 (<OS>-<arch> artifact naming), #780 / #796
  (Windows DPI + comctl32 manifest), #784 (FHS share/amule paths).

- Bug Fixes & Stability: post-#744 fixes including EC notification
  leak (#797), big-library scaling (#736, #840 superseding #728),
  amulegui ghost entries (#810, #819, #841, #824, #830, #760),
  PartFile early hash (#762), server protocol fixes (#835, #788,
  #721, #787), crypto stream UB (#779), UAF prevention (#756),
  Kad rotation (#795, #799/#805), GTK warning silencing (#833,
  #826/#836), and the clang-tidy worklist (#770, #772-#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (#753/#754/#776), Galician (#763), Slovenian (#771), pt-BR
  (#768/#775/#812), French (#811), plus man-page tooling for
  date+version drift (#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (#817/#818/#821/#828/#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with #746-#845 + #841.
got3nks added a commit to got3nks/amule that referenced this pull request Jul 31, 2026
The tree has no Java left. Its last .java file was the generated
ECCodes.java, removed in e86646c along with the other committed EC
headers, so the java-kotlin job has had nothing to scan since and fails on
every run.

Leaves actions, c-cpp and python, which is what the repository actually
contains.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants