Lift per-peer download throughput (Windows 4.3×, beats eMule 0.70b) - #484
Merged
mrjimenez merged 4 commits intoApr 27, 2026
Merged
Conversation
Replace the wait_read + sync read_some + drain-loop pattern with async_read_some + a 256 KB buffer. Maps to WSARecv on Windows (IOCP-native) and to the existing reactor read op on Linux/macOS (epoll/kqueue). Removes the drain loop that the previous edge- triggered scheme needed and the MSG_PEEK spurious-wakeup detection that came with it.
Background. The wire-level OP_REQUESTPARTS / OP_REQUESTPARTS_I64 pair carries exactly 3 <start,end> offset entries — see UploadClient.cpp ProcessRequestPartsPacket, which unconditionally reads three pairs and silently drops anything beyond. Stuffing more pairs into one packet makes the sender parse garbage; on a fresh peer that means it never issues OP_ACCEPTUPLOADREQ and the receiver hangs in "On queue". Until now, m_MaxBlockRequests served as both the local in-flight cap and the per-packet block count, so any value above STANDARD_BLOCKS_REQUEST=3 broke the wire format outright. The wxASSERT at the top of the legacy build path was the protocol invariant guard, but Release builds compile asserts out so it never caught the breakage. Pattern. Mirror eMule 0.70 srchybrid/DownloadClient.cpp::SendBlockRequests. Treat m_MaxBlockRequests as the local pending-list depth and emit one 3-block packet per call to SendBlockRequests; a new fQueued bit on Pending_Block_Struct distinguishes "in our local queue, not yet asked over the wire" from "already in a packet, sender owes us bytes". The caller's existing re-invocation chain (after each OP_SENDINGPART, on OP_ACCEPTUPLOADREQ, etc.) issues further packets until every pending slot is queued, which lifts the legacy peers' steady-state in-flight count above 3 the same way eMule does. For the ED2Kv2 path (gated on GetVBTTags(), which no real peer advertises) the packet still carries up to m_MaxBlockRequests blocks inline, but it now also walks only unqueued entries so the two paths share the same fQueued bookkeeping. This commit is a behaviour-equivalent refactor on its own. m_MaxBlockRequests remains STANDARD_BLOCKS_REQUEST = 3 in steady state (the VBT-gated ramp at the top of the function never fires in practice), so we still send 3 unqueued blocks in one packet, which is exactly what master sends. A follow-up replaces the VBT ramp with rate-based scaling and lifts the cap above 3, which now is safe.
wxMutex on Linux/macOS wraps pthread_mutex. std::mutex on libstdc++ takes the same path (PTHREAD_MUTEX_NORMAL by default), so the swap is neutral there. On Windows MinGW (clang/llvm libc++ and gcc libstdc++ alike), std::mutex is implemented over SRWLOCK while wxMutex is layered over CRITICAL_SECTION. SRWLOCK is the lighter primitive — single Interlocked compare-exchange for uncontended acquire vs the heavier spinlock-then-wait dance of CRITICAL_SECTION — so on the platform where amuled has been measurably slow this is a small uncontested win. All 17 wxMutexLocker call sites for m_sendLocker updated to std::lock_guard<std::mutex>. m_sendLocker is per-socket and never re-entered on the same thread so the non-recursive variant is correct. Build verified clean on macOS.
… transition
UploadBandwidthThrottler's main loop has an adaptive backoff
(extraSleepTime *= 5 per tick that didn't send any bytes, capped at
1 sec) that lets the thread doze when nothing has been queued.
Steady-state downloads enqueue OP_REQUESTPARTS roughly every 10-20 ms,
so the control queue is empty between calls and the throttler easily
ramps its sleep into the 5-25 ms range. When SendBlockRequests then
queues the next request, the throttler is asleep — the packet sits
in m_TempControlQueue_list until the next WaitTimeout returns. That
delay lives in the peer's request→response loop and on Windows (where
the wx-side wake hop is already several hundred microseconds) it was
the dominant per-stream throughput cap.
NewUploadDataAvailable() already exposes the right wake mechanism;
the disk I/O thread uses it whenever a fresh chunk lands on a socket
queue. Mirror that on the control-packet path, gated to the
empty→non-empty transition so bursty SendBlockRequests fan-out
collapses into a single signal (matches the CBatchDrainNotifier
pattern — one wake per drain cycle, not per producer add).
Build verified clean on macOS. Three-platform 120 s validation
against an aMule-fiber peer:
* Mac: peak 61.88 MB/s
* Linux ARM: peak 56.95 MB/s
* Windows: peak 30.36 MB/s (vs. 11 MB/s baseline, vs. 22 MB/s
eMule on the same hardware/peer)
5 tasks
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
May 5, 2026
Two tables at the top of the Performance section in 3.0.0:
1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download
throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over
LAN, 30 GB file:
macOS: 0.35 -> 135 MB/s (381x)
Linux: 0.34 -> 117 MB/s (345x)
Windows: 0.36 -> 39 MB/s (107x)
Plus a per-platform seeder/leecher contribution split that shows
the seeder-side fix (amule-project#451) dominates and the leecher-side stack
(amule-project#454/amule-project#484/amule-project#491) adds another 4.5-5.8x on top.
2. vs eMule 0.70b on Windows — same UTM hardware, both directions:
Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x)
Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x)
Methodology footnote on the table since eMule has no remote-
control protocol and is measured leecher-side via aMule's EC
channel on the receiving box.
Highlights line 16 also updated to mention both directions of the
eMule comparison instead of just download.
The existing per-PR benches in #### Upload / #### Download stay
unchanged — those are PR-specific numbers documenting how each
single change contributed; the new tables document the end-to-end
2.3.3-to-3.0.0 user experience.
Numbers measured with the bench-matrix.sh dispatcher
(scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2
manual aMule-vs-eMule runs. Raw logs in
bench-results/bench-matrix-20260505T141651Z/.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
May 5, 2026
Two tables at the top of the Performance section in 3.0.0:
1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download
throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over
LAN, 30 GB file:
macOS: 0.35 -> 135 MB/s (381x)
Linux: 0.34 -> 117 MB/s (345x)
Windows: 0.36 -> 39 MB/s (107x)
Plus a per-platform seeder/leecher contribution split that shows
the seeder-side fix (amule-project#451) dominates and the leecher-side stack
(amule-project#454/amule-project#484/amule-project#491) adds another 4.5-5.8x on top.
2. vs eMule 0.70b on Windows — same UTM hardware, both directions:
Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x)
Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x)
Methodology footnote on the table since eMule has no remote-
control protocol and is measured leecher-side via aMule's EC
channel on the receiving box.
Highlights line 16 also updated to mention both directions of the
eMule comparison instead of just download.
The existing per-PR benches in #### Upload / #### Download stay
unchanged — those are PR-specific numbers documenting how each
single change contributed; the new tables document the end-to-end
2.3.3-to-3.0.0 user experience.
Numbers measured with the bench-matrix.sh dispatcher
(scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2
manual aMule-vs-eMule runs. Raw logs in
bench-results/bench-matrix-20260505T141651Z/.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
May 5, 2026
Two tables at the top of the Performance section in 3.0.0:
1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download
throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over
LAN, 30 GB file:
macOS: 0.35 -> 135 MB/s (381x)
Linux: 0.34 -> 117 MB/s (345x)
Windows: 0.36 -> 39 MB/s (107x)
Plus a per-platform seeder/leecher contribution split that shows
the seeder-side fix (amule-project#451) dominates and the leecher-side stack
(amule-project#454/amule-project#484/amule-project#491) adds another 4.5-5.8x on top.
2. vs eMule 0.70b on Windows — same UTM hardware, both directions:
Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x)
Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x)
Methodology footnote on the table since eMule has no remote-
control protocol and is measured leecher-side via aMule's EC
channel on the receiving box.
Highlights line 16 also updated to mention both directions of the
eMule comparison instead of just download.
The existing per-PR benches in #### Upload / #### Download stay
unchanged — those are PR-specific numbers documenting how each
single change contributed; the new tables document the end-to-end
2.3.3-to-3.0.0 user experience.
Numbers measured with the bench-matrix.sh dispatcher
(scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2
manual aMule-vs-eMule runs. Raw logs in
bench-results/bench-matrix-20260505T141651Z/.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
May 5, 2026
Two tables at the top of the Performance section in 3.0.0:
1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download
throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over
LAN, 30 GB file:
macOS: 0.35 -> 135 MB/s (381x)
Linux: 0.34 -> 117 MB/s (345x)
Windows: 0.36 -> 39 MB/s (107x)
Plus a per-platform seeder/leecher contribution split that shows
the seeder-side fix (amule-project#451) dominates and the leecher-side stack
(amule-project#454/amule-project#484/amule-project#491) adds another 4.5-5.8x on top.
2. vs eMule 0.70b on Windows — same UTM hardware, both directions:
Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x)
Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x)
Methodology footnote on the table since eMule has no remote-
control protocol and is measured leecher-side via aMule's EC
channel on the receiving box.
Highlights line 16 also updated to mention both directions of the
eMule comparison instead of just download.
The existing per-PR benches in #### Upload / #### Download stay
unchanged — those are PR-specific numbers documenting how each
single change contributed; the new tables document the end-to-end
2.3.3-to-3.0.0 user experience.
Numbers measured with the bench-matrix.sh dispatcher
(scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2
manual aMule-vs-eMule runs. Raw logs in
bench-results/bench-matrix-20260505T141651Z/.
mrjimenez
pushed a commit
that referenced
this pull request
May 6, 2026
Two tables at the top of the Performance section in 3.0.0:
1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download
throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over
LAN, 30 GB file:
macOS: 0.35 -> 135 MB/s (381x)
Linux: 0.34 -> 117 MB/s (345x)
Windows: 0.36 -> 39 MB/s (107x)
Plus a per-platform seeder/leecher contribution split that shows
the seeder-side fix (#451) dominates and the leecher-side stack
(#454/#484/#491) adds another 4.5-5.8x on top.
2. vs eMule 0.70b on Windows — same UTM hardware, both directions:
Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x)
Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x)
Methodology footnote on the table since eMule has no remote-
control protocol and is measured leecher-side via aMule's EC
channel on the receiving box.
Highlights line 16 also updated to mention both directions of the
eMule comparison instead of just download.
The existing per-PR benches in #### Upload / #### Download stay
unchanged — those are PR-specific numbers documenting how each
single change contributed; the new tables document the end-to-end
2.3.3-to-3.0.0 user experience.
Numbers measured with the bench-matrix.sh dispatcher
(scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2
manual aMule-vs-eMule runs. Raw logs in
bench-results/bench-matrix-20260505T141651Z/.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Jul 20, 2026
…ct#530) The Directories preferences panel was inert in amuleGUI: its tree browses the wrong filesystem when the core is remote, and nothing it produced reached the core (the shared-folder lists are not in the EC preferences packet, and SaveSharedFolders() is compiled out under CLIENT_GUI), while the reload sent alongside made the core rescan its own unchanged config — so it looked applied. Adds EC_OP_GET/SET_SHARED_DIRS carrying the explicit and recursive roots, negotiated via EC_TAG_CAN_SHAREDDIRS_CONFIG. The core validates each path, applies and persists the valid ones, rescans, and reports refused paths with a locale-neutral reason; the union file is refreshed alongside the intent lists so the rescan cannot trim the new roots back out. amuleGUI replaces the tree with a Path/Recursive list editor owned by CPreferencesRem, which outlives the dialog. The monolithic tree now tracks the roots it was painted from and rebuilds only when they move, and its pending-edit flag is cleared at session end instead of latching for the dialog lifetime. muuli_wdr.cpp moves from the muleappgui static library into GUI_SOURCES so each executable compiles it with its own defines; it was previously built once without CLIENT_GUI, making every such branch in it dead code. Closes amule-project#484.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lifts per-peer download throughput across every platform we ship for. Windows is the headline (10 MB/s → 42 MB/s, ≈4×, beating eMule 0.70b on the same hardware) but macOS and Ubuntu ARM also roughly double from the same fix — the throttler-wake stall it removes was platform-agnostic, just dominant on Windows because wx event delivery there is already slow.
Four commits, all small, building on top of upstream
master:51dfb4366async_read_some(IOCP-native on Windows)122186711m_MaxBlockRequestsfrom per-packet block count2e2221b24wxMutex m_sendLockerwithstd::mutex65f9b31efm_newDataConditionon empty→non-empty control-queue transitionThe single most impactful commit is the throttler wake (
65f9b31ef) — bisection showed it carries essentially the entire perf gain on its own. The other three are small adjacent improvements: the IOCP-native read path is the right fast path on Windows, thewxMutex→std::mutexswap putsm_sendLockeron SRWLOCK on MinGW, and decouplingm_MaxBlockRequestsfrom the per-packet count restores the in-flight ceiling that the legacy 3-blocks-per-packet eD2k coupling clamped.Root cause
UploadBandwidthThrottler::Entry()runs an adaptive backoff: every loop iteration that didn't actually transmit any bytes growsextraSleepTime *= 5, capped at 1 sec. That is great for idle quiescence but nasty in steady-state downloads, where the control queue is empty between consecutiveOP_REQUESTPARTSenqueues (roughly every 10–25 ms at typical eD2k block-completion cadence). The throttler easily ramps its sleep into the 5–25 ms range. WhenSendBlockRequeststhen queues the next request, the throttler is dozing — the packet sits inm_TempControlQueue_listuntil the nextWaitTimeoutreturns. That delay lives squarely in the peer's request→response loop and was the dominant per-stream throughput cap on Windows (where the wx-side wake hop is already several hundred microseconds).NewUploadDataAvailable()already exposes the right wake mechanism for the disk I/O thread when fresh chunks land on a socket queue. The fix mirrors it on the control-packet path, gated to the empty→non-empty transition so burstySendBlockRequestsfan-out collapses into a single signal (matches the existingCBatchDrainNotifieridiom — one wake per drain cycle, not per producer add).Benchmark — single eD2k peer, identical hardware/network
Test rig: an aMule seeder on a separate LAN host (port 4662) plus a leech on each of macOS (Apple Silicon), Windows 11 ARM (under UTM), and Ubuntu ARM (also under UTM). 5 GB testfile, single source (
Sources 1(1)). 90 s ramp on a host-quiescent machine (no concurrent compiles), peak speed reported. Both ends always run the same revision — baseline = upstreammasteron all four machines, post-PR = this branch on all four. eMule 0.70b reference captured with a tcpdump during a real eMule download from the same seeder; throughput computed from the TCP conversation summary.For reference on the same Windows hardware/peer:
Windows aMule with this PR is ≈1.9× eMule 0.70b on the same machine and peer.
How we got there
A heavy bisection session against the same seeder. The 4 commits in this PR are what survived after each iteration of "drop a commit, re-test on Mac + Windows host-quiescent". Highlights of what we explicitly tried and dropped (kept for the record):
m_sendLockerout of the do-while inOnReceive— broke the seeder severely (constant ~1.9 MB/s) by starving the throttler thread ofm_sendLockerwhile the main thread held it acrossPacketReceived. Reverted.READ_CHUNK64 KB → 2 MB — reduced kernel-side completion frequency on Windows but no measurable throughput delta.HandleRead(eMule pattern) — same: collapsed many small completions into one but no measurable throughput delta.All of those changes are stable and well-described in the working branches if anyone wants them later, but each one was unable to demonstrate a measurable throughput gain on top of the throttler wake gate.
Test plan
aMuleD GIT compiled with wxBase(OSX Cocoa) v3.3.2 and Boost 1.90): 90 s ramp against the rev-aligned seeder, peak 145 MB/s.aMuleD GIT compiled with wxBase(MSW) v3.2.10 and Boost 1.90): 90 s ramp, peak 42 MB/s.aMuleD GIT compiled with wxBase(GTK3) v3.2.8 and Boost 1.88): 90 s ramp on this branch, peak 115 MB/s.cmake -USVNDATEis the workaround for the build-system bug fixed separately in cmake: stop freezing the SVNDATE banner string at first configure #483).Notes for review
m_newDataMutex/m_newDataConditionpair already exists for the disk-I/O wake path; this commit does not introduce new synchronization primitives, just routes the control-packet path through the same wake mechanism.m_tempQueueLockeris taken first and released beforem_newDataMutex, matchingNewUploadDataAvailable(); no new ordering risk.m_sendLockeris per-socket and never re-entered on the same thread, so non-recursivestd::mutexis safe (we explicitly verified this — an earlier hoist patch neededstd::recursive_mutex, which we then abandoned along with the hoist).