Skip to content

ec: move ZLIB locality decision from server peer-IP to client tag + user override - #840

Merged
mrjimenez merged 1 commit into
amule-project:masterfrom
got3nks:feat/ec-prefer-no-zlib
Jun 4, 2026
Merged

ec: move ZLIB locality decision from server peer-IP to client tag + user override#840
mrjimenez merged 1 commit into
amule-project:masterfrom
got3nks:feat/ec-prefer-no-zlib

Conversation

@got3nks

@got3nks got3nks commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Continues the discussion on #728 (thanks @danim7 for raising the WireGuard case).

Why

#728 made the daemon skip ZLIB on per-packet basis when the EC peer's IP looked "local" (loopback / RFC1918 / link-local). The check lived on the server side and was based on the peer-IP it saw on the socket, which misclassifies common topologies — most notably WireGuard / Tailscale tunnel endpoints that resolve to RFC1918 addresses but whose underlying transit is the public Internet. Those connections lost ZLIB compression silently and paid the wire-size cost.

What

Move the locality decision to the client, which is the only side that knows the IP it actually dialed.

  • New tag EC_TAG_PREFER_NO_ZLIB (0x0014) — a preference, not a capability. EC_TAG_CAN_ZLIB stays as the always-on capability so the oversize-payload fallback (>EC_MAX_UNCOMPRESSED forces ZLIB regardless of preference at ECSocket.cpp:797) keeps working when the receiver gate would otherwise reject the packet.
  • CRemoteConnect::ConnectToCore resolves the dialed host and sets m_preferNoZlib when it points at a loopback / RFC1918 / link-local address. CECLoginPacket ctor takes a new preferNoZlib param and emits the tag when set.
  • ExternalConn drops the peer-IP block in CECServerSocket::Authenticate; SetLocalPeer(true) now fires only when the client sent EC_TAG_PREFER_NO_ZLIB.
  • User override: amulegui dialog gets a "Force ZLIB compression" checkbox (persists under "Remember those settings" to /EC/ForceZLIB); amulecmd / amuleweb get --force-zlib and the same config key in remote.conf. When set, the locality check is skipped and the tag is never sent — server compresses as if we were remote. Use case: VPN-as-LAN topologies.

Backward compatibility

Verified

  • macOS build: clean
  • Linux build (Debug, x86_64): clean
  • amulecmdamuled on 127.0.0.1, default: server logs EC peer 127.0.0.1 asked to skip ZLIB (loopback/LAN hint), ZLIB capability advertised
  • amulecmd --force-zlibamuled on 127.0.0.1: NO "asked to skip ZLIB" line, ZLIB capability advertised — server compresses as remote
  • amulegui checkbox UX (pending — same wire behaviour as --force-zlib)

amule-project#728 made the daemon skip ZLIB on per-packet basis when the EC peer's
IP looked "local" (loopback / RFC1918 / link-local). The check lived on
the server side and was based on the peer-IP it saw on the socket,
which misclassifies common topologies — most importantly WireGuard /
Tailscale tunnel endpoints that resolve to RFC1918 addresses but whose
underlying transit is the public Internet. Those connections lost ZLIB
compression silently and paid the wire-size cost.

Move the locality decision to the client, which is the only side that
knows the IP it actually dialed:

  - New tag EC_TAG_PREFER_NO_ZLIB (0x0014) — a *preference*, not a
    capability. EC_TAG_CAN_ZLIB stays as the always-on capability so
    the oversize-payload fallback (>EC_MAX_UNCOMPRESSED forces ZLIB
    regardless of preference, see ECSocket.cpp:797) still kicks in
    when the receiver-gate would otherwise reject the packet.

  - CRemoteConnect::ConnectToCore resolves the dialed host and sets
    m_preferNoZlib when it points at a loopback / RFC1918 / link-local
    address. CECLoginPacket takes a new preferNoZlib param and emits
    the tag when set.

  - ExternalConn drops the peer-IP block in CECServerSocket::
    Authenticate; SetLocalPeer(true) now fires only when the client
    sent EC_TAG_PREFER_NO_ZLIB. Old clients omit the tag and get the
    pre-amule-project#728 always-ZLIB behaviour. Old servers ignore the unknown
    tag and treat the connection as remote (also pre-amule-project#728).

  - User override: amulegui dialog gets a "Force ZLIB compression"
    checkbox (persists under "Remember those settings" to
    /EC/ForceZLIB); amulecmd / amuleweb get --force-zlib and the
    same config key in remote.conf. When set, the locality check is
    skipped and the tag is never sent, so the server compresses as
    if we were remote. Use case: VPN-as-LAN topologies where the
    dialed IP looks local but transit isn't.

ECSocket.cpp's per-packet send logic (m_isLocalPeer / EC_FLAG_ZLIB /
oversize fallback) is unchanged; only the meaning of m_isLocalPeer's
input changes.

Closes the discussion thread on amule-project#728 (danim7 raised the WireGuard case).
@got3nks

got3nks commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

@ngosang we should update the docs with the new GUI + remote.conf settings.

@got3nks

got3nks commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

@mrjimenez all tested and ready to merge.

@mrjimenez
mrjimenez merged commit b6f91bb into amule-project:master Jun 4, 2026
7 checks passed
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
…ndex

Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(amule-project#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg
  amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796
  (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths).

- Bug Fixes & Stability: post-amule-project#744 fixes including EC notification
  leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728),
  amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760),
  PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788,
  amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756),
  Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833,
  amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR
  (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for
  date+version drift (amule-project#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
Captures translatable strings introduced after the previous po
refresh (post-amule-project#747 wave). Notable new entries:

  - "Connection failed. Please check the host, port, and password."
    (amule-project#841 — amulegui retry-on-connect-failure)
  - "Force ZLIB compression" (amule-project#840 — EC client-tag locality override)
  - "Follow symbolic links in shared folders" (amule-project#809)
  - "Connection Type:" (amule-project#824 — ED2K Info readability)
  - "Documentation: https://amule-org.github.io/docs " (amule-project#801 — wiki
    URL migration)

Generated with `scripts/update-po.sh` from a working tree that has
PR amule-project#841 applied locally, so the new amulegui retry-loop strings
are included even though amule-project#841 itself is not in this branch — when
amule-project#841 lands on master, the .po files will already reflect its
strings without needing a second regeneration pass.

The remaining churn is line-number realignment across the ~38
language files from a month of source-code drift.
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
…ndex

Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(amule-project#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg
  amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796
  (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths).

- Bug Fixes & Stability: post-amule-project#744 fixes including EC notification
  leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728),
  amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760),
  PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788,
  amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756),
  Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833,
  amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR
  (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for
  date+version drift (amule-project#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
Captures translatable strings introduced after the previous po
refresh (post-amule-project#747 wave). Notable new entries:

  - "Connection failed. Please check the host, port, and password."
    (amule-project#841 — amulegui retry-on-connect-failure)
  - "Force ZLIB compression" (amule-project#840 — EC client-tag locality override)
  - "Follow symbolic links in shared folders" (amule-project#809)
  - "Connection Type:" (amule-project#824 — ED2K Info readability)
  - "Documentation: https://amule-org.github.io/docs " (amule-project#801 — wiki
    URL migration)

Generated with `scripts/update-po.sh` from a working tree that has
PR amule-project#841 applied locally, so the new amulegui retry-loop strings
are included even though amule-project#841 itself is not in this branch — when
amule-project#841 lands on master, the .po files will already reflect its
strings without needing a second regeneration pass.

The remaining churn is line-number realignment across the ~38
language files from a month of source-code drift.
mrjimenez pushed a commit that referenced this pull request Jun 4, 2026
Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for #785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), #794 (.dmg
  amuleweb path), #789 (<OS>-<arch> artifact naming), #780 / #796
  (Windows DPI + comctl32 manifest), #784 (FHS share/amule paths).

- Bug Fixes & Stability: post-#744 fixes including EC notification
  leak (#797), big-library scaling (#736, #840 superseding #728),
  amulegui ghost entries (#810, #819, #841, #824, #830, #760),
  PartFile early hash (#762), server protocol fixes (#835, #788,
  #721, #787), crypto stream UB (#779), UAF prevention (#756),
  Kad rotation (#795, #799/#805), GTK warning silencing (#833,
  #826/#836), and the clang-tidy worklist (#770, #772-#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (#753/#754/#776), Galician (#763), Slovenian (#771), pt-BR
  (#768/#775/#812), French (#811), plus man-page tooling for
  date+version drift (#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (#817/#818/#821/#828/#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with #746-#845 + #841.
mrjimenez pushed a commit that referenced this pull request Jun 4, 2026
Captures translatable strings introduced after the previous po
refresh (post-#747 wave). Notable new entries:

  - "Connection failed. Please check the host, port, and password."
    (#841 — amulegui retry-on-connect-failure)
  - "Force ZLIB compression" (#840 — EC client-tag locality override)
  - "Follow symbolic links in shared folders" (#809)
  - "Connection Type:" (#824 — ED2K Info readability)
  - "Documentation: https://amule-org.github.io/docs " (#801 — wiki
    URL migration)

Generated with `scripts/update-po.sh` from a working tree that has
PR #841 applied locally, so the new amulegui retry-loop strings
are included even though #841 itself is not in this branch — when
#841 lands on master, the .po files will already reflect its
strings without needing a second regeneration pass.

The remaining churn is line-number realignment across the ~38
language files from a month of source-code drift.
@got3nks
got3nks deleted the feat/ec-prefer-no-zlib branch June 5, 2026 22:35
@ngosang

ngosang commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

@got3nks Just a question. How this affects the amuleweb?
I always experience a really slow load of the "shared files tab" in amuleweb because I have +10k and there is no pagination. This change could improve that? Should I enable it by default in my Docker image?

@danim7

danim7 commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

@got3nks Just a question. How this affects the amuleweb? I always experience a really slow load of the "shared files tab" in amuleweb because I have +10k and there is no pagination. This change could improve that? Should I enable it by default in my Docker image?

This PR concerns the transfer of data between amuled and amuleweb over the EC protocol. If both amuled and amuleweb run on the same docker image or server, you should not enable --force-zlib, because it will only waste CPU for your use case. This PR is useful when amuled and amuleweb connect over a slow link that looks local because of using LAN IP addresses.

Your performance issue on amuleweb for the +10k library looks like a different issue, maybe @got3nks can help debug it.

@got3nks

got3nks commented Jun 6, 2026

Copy link
Copy Markdown
Contributor Author

Quick way to split the bottleneck — Chrome / Firefox DevTools, with one specific number being the answer:

1. Profile the request in Network panel (F12 → Network):

  • Reload the Shared Files tab
  • Click on the main HTML request (e.g. amuleweb?op=sharedfiles)
  • Look at the Timing breakdown:

2. Profile rendering in Performance panel (F12 → Performance):

Once those three numbers are in hand the answer falls out:

Where time goes What helps
> 50 % in Waiting (TTFB) EC fetch + amuleweb HTML render. PR #840 skips ZLIB on local-peer EC so the amuleweb → amuled round-trip drops (skip the compress on the daemon side + skip the decompress on amuleweb's side). Real speedup on big sharesets when both sit on the same host (which is your Docker case).
Mostly Content Download Only the HTTP-side compression matters. Unrelated to #840.
Mostly browser layout/paint after the response Genuinely the 10 k DOM rows. Needs server-side pagination / virtualized scrolling in amuleweb's template, not an EC tweak.

On the Docker default question: for the always-co-located amuled + amuleweb pair you ship, the override that disables ZLIB on local-peer EC is exactly the case the PR was tuned for — both sides save CPU on every fetch and the localhost transfer cost is negligible. Worth enabling by default in your image; the only reason not to ship it on by default upstream was the "remote amuleweb via a slow link" minority case, which doesn't apply in your container.

If you can run the DevTools capture once and paste the Waiting / Content Download / browser-render numbers, I can tell you specifically how much of your slow load #840 would shave off vs how much is locked-in browser cost.

@ngosang

ngosang commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

amuleweb is working much better in amule 3.0.0

@got3nks

got3nks commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

amuleweb is working much better in amule 3.0.0

Happy to hear that. EC protocol had a lot of improvements, this is just one of them.

got3nks added a commit to got3nks/amule that referenced this pull request Aug 7, 2026
Closes amule-project#840. Servers cap how many of a user's shared files they will index --
some at 9,000, others at 100,000 or more -- and nothing in aMule surfaced it,
so a user sharing 37,000 files had no way to tell whether a server was
publishing all of them.

Adds Soft Files, Hard Files and Max Users to the server list. All three were
already parsed from server.met, kept on CServer and refreshed by the periodic
UDP status ping that fills Users and Files; only the display was missing. Soft
and Hard files are now streamed over EC as well, so the remote GUI has them
too -- Max Users already was.

Zero renders blank, not "0". These fields only arrive once a status reply has
come back, and the reply is parsed progressively by size: Max Users needs 16
bytes, the two limits 24. A server that sends a short reply, or has not
answered yet, advertises nothing -- and showing that as a limit of zero would
be a lie about the server rather than an absence of data.

Three fixes to the column layer came out of building this. Two predate it:

  - Drag-resizing a column was never persisted. The idle poll that stands in
    for wxDataViewCtrl's missing end-of-drag event called a virtual that
    defaulted to doing nothing, so a width survived only if the user happened
    to sort or toggle a column afterwards. It now saves, for every list on the
    dataview layer; CSearchListCtrl's override chains to the base so it keeps
    syncing widths across search tabs as well.

  - The remote GUI never sized its server columns to their contents at all.
    The core fits them when a bulk load finishes, via Notify_ServerThaw() from
    CServerList -- a path amulegui does not take, since its list arrives over
    EC. It now fits once, after the first populated update: refitting on every
    update would undo a width the user had dragged.

And one this work introduced: the column ids have to stay in the order the
columns are appended, because FitColumnsToContent() walks a single index and
uses it as both a model column and a view position. Moving Max Users next to
Users without renumbering sized every column after it against a different
one's content, which read as columns collapsing to nothing. The ids now match
the append order, and the constraint is written down next to them -- as is the
reason the two flag columns must stay last, being the pair the remote GUI does
not append.
mrjimenez pushed a commit to mrjimenez/amule that referenced this pull request Aug 8, 2026
…roject#841)

Closes amule-project#840. Servers cap how many of a user's shared files they will index --
some at 9,000, others at 100,000 or more -- and nothing surfaced it, so a user
sharing 37,000 files could not tell whether a server was publishing all of
them.

Adds Soft Files, Hard Files and Max Users to the server list. All three were
already parsed from server.met, kept on CServer and refreshed by the periodic
UDP status ping that fills Users and Files; only the display was missing. Soft
and hard files are streamed over EC so the remote GUI has them too.

Zero renders blank rather than "0": these arrive only once a status reply comes
back, and the reply is parsed progressively by size, so a server that has not
answered advertises nothing. Showing that as a limit of zero would be a claim
about the server rather than an absence of data.

The wire-flag columns are now offered in both binaries as well (requested by
danim7): the flags are streamed like the limits, the columns are appended,
registered and rendered unconditionally, and hidden-by-default keys on the
build type rather than on which binary is running.

Several column-layer defects came out of building it. Two predate this work:
drag-resizing was never persisted, because the idle poll standing in for
wxDataViewCtrl's missing end-of-drag event called a virtual that did nothing;
and the remote GUI never sized its columns to content at all, since the core's
bulk-load notification is a path it does not take. Both are fixed, and the
auto-fit is gated on the profile having no stored widths so it cannot overwrite
a width the user dragged -- a clobber that was harmless only while nothing was
saved.

Also: column ids must match the order columns are appended, since the fit walks
one index as both a model column and a view position -- now asserted rather
than documented; and the header show/hide menu was limited to 15 columns, which
silently dropped the 16th.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants