ec: move ZLIB locality decision from server peer-IP to client tag + user override - #840
Conversation
amule-project#728 made the daemon skip ZLIB on per-packet basis when the EC peer's IP looked "local" (loopback / RFC1918 / link-local). The check lived on the server side and was based on the peer-IP it saw on the socket, which misclassifies common topologies — most importantly WireGuard / Tailscale tunnel endpoints that resolve to RFC1918 addresses but whose underlying transit is the public Internet. Those connections lost ZLIB compression silently and paid the wire-size cost. Move the locality decision to the client, which is the only side that knows the IP it actually dialed: - New tag EC_TAG_PREFER_NO_ZLIB (0x0014) — a *preference*, not a capability. EC_TAG_CAN_ZLIB stays as the always-on capability so the oversize-payload fallback (>EC_MAX_UNCOMPRESSED forces ZLIB regardless of preference, see ECSocket.cpp:797) still kicks in when the receiver-gate would otherwise reject the packet. - CRemoteConnect::ConnectToCore resolves the dialed host and sets m_preferNoZlib when it points at a loopback / RFC1918 / link-local address. CECLoginPacket takes a new preferNoZlib param and emits the tag when set. - ExternalConn drops the peer-IP block in CECServerSocket:: Authenticate; SetLocalPeer(true) now fires only when the client sent EC_TAG_PREFER_NO_ZLIB. Old clients omit the tag and get the pre-amule-project#728 always-ZLIB behaviour. Old servers ignore the unknown tag and treat the connection as remote (also pre-amule-project#728). - User override: amulegui dialog gets a "Force ZLIB compression" checkbox (persists under "Remember those settings" to /EC/ForceZLIB); amulecmd / amuleweb get --force-zlib and the same config key in remote.conf. When set, the locality check is skipped and the tag is never sent, so the server compresses as if we were remote. Use case: VPN-as-LAN topologies where the dialed IP looks local but transit isn't. ECSocket.cpp's per-packet send logic (m_isLocalPeer / EC_FLAG_ZLIB / oversize fallback) is unchanged; only the meaning of m_isLocalPeer's input changes. Closes the discussion thread on amule-project#728 (danim7 raised the WireGuard case).
|
@ngosang we should update the docs with the new GUI + remote.conf settings. |
|
@mrjimenez all tested and ready to merge. |
…ndex Adds 55+ merged PRs to the 3.0.0 changelog since the last update (amule-project#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796 (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths). - Bug Fixes & Stability: post-amule-project#744 fixes including EC notification leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728), amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760), PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788, amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756), Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833, amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774). - Translations: late-cycle wave covering French/Turkish manpages (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for date+version drift (amule-project#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
Captures translatable strings introduced after the previous po refresh (post-amule-project#747 wave). Notable new entries: - "Connection failed. Please check the host, port, and password." (amule-project#841 — amulegui retry-on-connect-failure) - "Force ZLIB compression" (amule-project#840 — EC client-tag locality override) - "Follow symbolic links in shared folders" (amule-project#809) - "Connection Type:" (amule-project#824 — ED2K Info readability) - "Documentation: https://amule-org.github.io/docs " (amule-project#801 — wiki URL migration) Generated with `scripts/update-po.sh` from a working tree that has PR amule-project#841 applied locally, so the new amulegui retry-loop strings are included even though amule-project#841 itself is not in this branch — when amule-project#841 lands on master, the .po files will already reflect its strings without needing a second regeneration pass. The remaining churn is line-number realignment across the ~38 language files from a month of source-code drift.
…ndex Adds 55+ merged PRs to the 3.0.0 changelog since the last update (amule-project#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796 (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths). - Bug Fixes & Stability: post-amule-project#744 fixes including EC notification leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728), amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760), PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788, amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756), Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833, amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774). - Translations: late-cycle wave covering French/Turkish manpages (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for date+version drift (amule-project#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
Captures translatable strings introduced after the previous po refresh (post-amule-project#747 wave). Notable new entries: - "Connection failed. Please check the host, port, and password." (amule-project#841 — amulegui retry-on-connect-failure) - "Force ZLIB compression" (amule-project#840 — EC client-tag locality override) - "Follow symbolic links in shared folders" (amule-project#809) - "Connection Type:" (amule-project#824 — ED2K Info readability) - "Documentation: https://amule-org.github.io/docs " (amule-project#801 — wiki URL migration) Generated with `scripts/update-po.sh` from a working tree that has PR amule-project#841 applied locally, so the new amulegui retry-loop strings are included even though amule-project#841 itself is not in this branch — when amule-project#841 lands on master, the .po files will already reflect its strings without needing a second regeneration pass. The remaining churn is line-number realignment across the ~38 language files from a month of source-code drift.
Adds 55+ merged PRs to the 3.0.0 changelog since the last update (#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for #785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), #794 (.dmg amuleweb path), #789 (<OS>-<arch> artifact naming), #780 / #796 (Windows DPI + comctl32 manifest), #784 (FHS share/amule paths). - Bug Fixes & Stability: post-#744 fixes including EC notification leak (#797), big-library scaling (#736, #840 superseding #728), amulegui ghost entries (#810, #819, #841, #824, #830, #760), PartFile early hash (#762), server protocol fixes (#835, #788, #721, #787), crypto stream UB (#779), UAF prevention (#756), Kad rotation (#795, #799/#805), GTK warning silencing (#833, #826/#836), and the clang-tidy worklist (#770, #772-#774). - Translations: late-cycle wave covering French/Turkish manpages (#753/#754/#776), Galician (#763), Slovenian (#771), pt-BR (#768/#775/#812), French (#811), plus man-page tooling for date+version drift (#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (#817/#818/#821/#828/#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with #746-#845 + #841.
Captures translatable strings introduced after the previous po refresh (post-#747 wave). Notable new entries: - "Connection failed. Please check the host, port, and password." (#841 — amulegui retry-on-connect-failure) - "Force ZLIB compression" (#840 — EC client-tag locality override) - "Follow symbolic links in shared folders" (#809) - "Connection Type:" (#824 — ED2K Info readability) - "Documentation: https://amule-org.github.io/docs " (#801 — wiki URL migration) Generated with `scripts/update-po.sh` from a working tree that has PR #841 applied locally, so the new amulegui retry-loop strings are included even though #841 itself is not in this branch — when #841 lands on master, the .po files will already reflect its strings without needing a second regeneration pass. The remaining churn is line-number realignment across the ~38 language files from a month of source-code drift.
|
@got3nks Just a question. How this affects the amuleweb? |
This PR concerns the transfer of data between amuled and amuleweb over the EC protocol. If both amuled and amuleweb run on the same docker image or server, you should not enable Your performance issue on amuleweb for the +10k library looks like a different issue, maybe @got3nks can help debug it. |
|
Quick way to split the bottleneck — Chrome / Firefox DevTools, with one specific number being the answer: 1. Profile the request in Network panel (F12 → Network):
2. Profile rendering in Performance panel (F12 → Performance):
Once those three numbers are in hand the answer falls out:
On the Docker default question: for the always-co-located If you can run the DevTools capture once and paste the Waiting / Content Download / browser-render numbers, I can tell you specifically how much of your slow load #840 would shave off vs how much is locked-in browser cost. |
|
amuleweb is working much better in amule 3.0.0 |
Happy to hear that. EC protocol had a lot of improvements, this is just one of them. |
Closes amule-project#840. Servers cap how many of a user's shared files they will index -- some at 9,000, others at 100,000 or more -- and nothing in aMule surfaced it, so a user sharing 37,000 files had no way to tell whether a server was publishing all of them. Adds Soft Files, Hard Files and Max Users to the server list. All three were already parsed from server.met, kept on CServer and refreshed by the periodic UDP status ping that fills Users and Files; only the display was missing. Soft and Hard files are now streamed over EC as well, so the remote GUI has them too -- Max Users already was. Zero renders blank, not "0". These fields only arrive once a status reply has come back, and the reply is parsed progressively by size: Max Users needs 16 bytes, the two limits 24. A server that sends a short reply, or has not answered yet, advertises nothing -- and showing that as a limit of zero would be a lie about the server rather than an absence of data. Three fixes to the column layer came out of building this. Two predate it: - Drag-resizing a column was never persisted. The idle poll that stands in for wxDataViewCtrl's missing end-of-drag event called a virtual that defaulted to doing nothing, so a width survived only if the user happened to sort or toggle a column afterwards. It now saves, for every list on the dataview layer; CSearchListCtrl's override chains to the base so it keeps syncing widths across search tabs as well. - The remote GUI never sized its server columns to their contents at all. The core fits them when a bulk load finishes, via Notify_ServerThaw() from CServerList -- a path amulegui does not take, since its list arrives over EC. It now fits once, after the first populated update: refitting on every update would undo a width the user had dragged. And one this work introduced: the column ids have to stay in the order the columns are appended, because FitColumnsToContent() walks a single index and uses it as both a model column and a view position. Moving Max Users next to Users without renumbering sized every column after it against a different one's content, which read as columns collapsing to nothing. The ids now match the append order, and the constraint is written down next to them -- as is the reason the two flag columns must stay last, being the pair the remote GUI does not append.
…roject#841) Closes amule-project#840. Servers cap how many of a user's shared files they will index -- some at 9,000, others at 100,000 or more -- and nothing surfaced it, so a user sharing 37,000 files could not tell whether a server was publishing all of them. Adds Soft Files, Hard Files and Max Users to the server list. All three were already parsed from server.met, kept on CServer and refreshed by the periodic UDP status ping that fills Users and Files; only the display was missing. Soft and hard files are streamed over EC so the remote GUI has them too. Zero renders blank rather than "0": these arrive only once a status reply comes back, and the reply is parsed progressively by size, so a server that has not answered advertises nothing. Showing that as a limit of zero would be a claim about the server rather than an absence of data. The wire-flag columns are now offered in both binaries as well (requested by danim7): the flags are streamed like the limits, the columns are appended, registered and rendered unconditionally, and hidden-by-default keys on the build type rather than on which binary is running. Several column-layer defects came out of building it. Two predate this work: drag-resizing was never persisted, because the idle poll standing in for wxDataViewCtrl's missing end-of-drag event called a virtual that did nothing; and the remote GUI never sized its columns to content at all, since the core's bulk-load notification is a path it does not take. Both are fixed, and the auto-fit is gated on the profile having no stored widths so it cannot overwrite a width the user dragged -- a clobber that was harmless only while nothing was saved. Also: column ids must match the order columns are appended, since the fit walks one index as both a model column and a view position -- now asserted rather than documented; and the header show/hide menu was limited to 15 columns, which silently dropped the 16th.
Continues the discussion on #728 (thanks @danim7 for raising the WireGuard case).
Why
#728 made the daemon skip ZLIB on per-packet basis when the EC peer's IP looked "local" (loopback / RFC1918 / link-local). The check lived on the server side and was based on the peer-IP it saw on the socket, which misclassifies common topologies — most notably WireGuard / Tailscale tunnel endpoints that resolve to RFC1918 addresses but whose underlying transit is the public Internet. Those connections lost ZLIB compression silently and paid the wire-size cost.
What
Move the locality decision to the client, which is the only side that knows the IP it actually dialed.
EC_TAG_PREFER_NO_ZLIB (0x0014)— a preference, not a capability.EC_TAG_CAN_ZLIBstays as the always-on capability so the oversize-payload fallback (>EC_MAX_UNCOMPRESSEDforces ZLIB regardless of preference atECSocket.cpp:797) keeps working when the receiver gate would otherwise reject the packet.CRemoteConnect::ConnectToCoreresolves the dialed host and setsm_preferNoZlibwhen it points at a loopback / RFC1918 / link-local address.CECLoginPacketctor takes a newpreferNoZlibparam and emits the tag when set.ExternalConndrops the peer-IP block inCECServerSocket::Authenticate;SetLocalPeer(true)now fires only when the client sentEC_TAG_PREFER_NO_ZLIB./EC/ForceZLIB); amulecmd / amuleweb get--force-zliband the same config key inremote.conf. When set, the locality check is skipped and the tag is never sent — server compresses as if we were remote. Use case: VPN-as-LAN topologies.Backward compatibility
Verified
amulecmd→amuledon 127.0.0.1, default: server logsEC peer 127.0.0.1 asked to skip ZLIB (loopback/LAN hint), ZLIB capability advertisedamulecmd --force-zlib→amuledon 127.0.0.1: NO "asked to skip ZLIB" line, ZLIB capability advertised — server compresses as remote--force-zlib)