upload: adaptive payload buffer and packet chunk size for fast connections - #444
Closed
got3nks wants to merge 2 commits into
Closed
upload: adaptive payload buffer and packet chunk size for fast connections#444got3nks wants to merge 2 commits into
got3nks wants to merge 2 commits into
Conversation
got3nks
force-pushed
the
adaptive-upload-buffer
branch
from
April 13, 2026 20:15
3028800 to
ab6af17
Compare
Buffer ~10 seconds worth of data per upload slot, clamped between
180 KiB (eMule minimum) and 16 MiB (upper bound):
payloadBufferLimit = clamp(GetUploadDatarate() * 10, 180 KiB, 16 MiB)
Compensates for aMule's synchronous disk I/O: the throttler can drain
the buffer between CreateNextBlockPackage() calls, so a larger lookahead
prevents socket starvation on fast slots. Intended as a temporary measure
until an async disk I/O thread (eMule-style) is implemented.
Scale chunk size with actual slot throughput, floor 10 KiB (stock aMule),
ceil 128 KiB:
chunkSize = clamp(GetUploadDatarate() / 8, 10 KiB, 128 KiB)
Slow slots keep small chunks for smooth flow; fast slots (>1 MB/s) get
128 KiB chunks to reduce per-packet header overhead and socket queue
churn.
got3nks
force-pushed
the
adaptive-upload-buffer
branch
from
April 13, 2026 20:16
ab6af17 to
de3ac51
Compare
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Apr 15, 2026
Scale upload packet chunk size with the per-slot upload datarate: min(max(datarate/8, 10240), 131072) At low speeds the default 10 KiB chunks keep latency low. At high speeds (e.g. LAN or fast peers) chunks grow up to 128 KiB, reducing per-packet overhead from ed2k framing, encryption, and syscalls. Benchmark on LAN (Docker, 64 MB/s upload cap): Before: 20 MB/s avg, 61706 chunks in 30s After: 59 MB/s avg, 13051 chunks in 30s (3x throughput) ref: amule-project#444
Contributor
Author
|
Superseded by #451, which adopts eMule's CUploadDiskIOThread approach end-to-end (background disk I/O + packet construction) and keeps the adaptive chunk sizing from this PR on top of it. Closing in favour of that. |
ngosang
pushed a commit
to ngosang/amule
that referenced
this pull request
Jul 12, 2026
amule-project#444) (amule-project#452) When the EC connection to the remote core dropped after startup (e.g. the machine slept or a VPN/SSH tunnel restarted and the socket died), amulegui showed "aMule has terminated probably" and exited, forcing a full restart + resync. It now reconnects in the background instead. Reconnect mechanics: - CLibSocket::ResetForReconnect() swaps a fresh asio impl onto the SAME CRemoteConnect, so the socket can be re-opened after a loss without recreating the object (every remote container pins its CRemoteConnect). CECSocket::ResetProtocolState() rewinds the packet-reassembly state machine so a mid-packet read left over from the drop can't misparse the reconnected session's first bytes. - On a post-startup loss the UI is frozen behind a modal reconnect dialog (attempt counter + countdown to the next try + "Abort and exit"); attempts run every 5 s with a 15 s per-attempt watchdog, until the connection is restored or the user aborts. - amuleweb / amulecmd keep their fail-fast _exit(1) on loss (the new paths are all gated on the GUI's m_notifier). Two fixes the reused socket needs to hand off cleanly: - Clear m_ErrorCode on a successful connect: the swap can leave a stale EBADF from an aborted read on the impl, and SocketRealError() would then make CECSocket::WritePacket refuse to send the login. - Swallow a LibSocketLost that arrives while a dial is still pending (EC_CONNECT_SENT): it's a stale event queued for the previous connection before the impl was swapped, and would otherwise abort the in-flight reconnect. A genuinely failed dial is caught by the connect-timeout watchdog instead. Resync is reconcile-in-place, not wipe+rebuild, so scroll and selection survive: - Polling resumes and the fresh full snapshot updates rows in place / adds new / prunes gone. A partial-update server won't re-emit FILE_REMOVED for files deleted while disconnected, so CKnownFilesRem forces a one-shot prune-by-absence on the first post-reconnect update (m_reconnectReconcile). The server / client / friend lists already prune every poll, so they self-heal with no special case. - The daemon keeps its RLE gap/part/req-status encoders per connection, so a reconnect restarts them from an empty baseline. The reused CKnownFile / CPartFile decoders (m_PartFileEncoderData, m_partStatus) are reset to match, or the first differential update would XOR against a stale buffer and paint garbage (all-red progress bars, wrong availability shading). - Large libraries stay responsive: the reconnect poll wraps the download + shared list ctrls in BeginBatchUpdate()/EndBatchUpdate() (one repaint + one sort, per-item sort suppressed) so a 10k+ resync doesn't hitch. No daemon or EC wire-protocol changes: reconnect works against a stock amuled.
ngosang
pushed a commit
to ngosang/amule
that referenced
this pull request
Jul 13, 2026
amule-project#444) (amule-project#465) Follow-up to amule-project#452, from two issues reported after a successful reconnect. 1. Empty download queue + shared file lists (restart required) The EC request FIFO (CRemoteConnect::m_req_fifo) assumes the core answers every request in FCFS order. A socket dropped mid-poll leaves the requests that were on the air unanswered, so their handlers linger in the FIFO. After the reconnect each reply pops the wrong (stale) handler: a stats reply routed to CKnownFilesRem drives its one-shot post-reconnect reconcile against an empty file set, and the absence-prune wipes the whole library. It is intermittent (depends how many requests were in flight at the drop) and only a full restart clears it, since a fresh process starts with an empty FIFO. - CRemoteConnect::DiscardRequestQueue() flushes the FIFO and zeroes the in-flight counter on reconnect, rewinding each orphaned handler's request state (CECPacketHandlerBase::AbortPendingRequest, overridden by CRemoteContainer to reset its request SM to IDLE) so a container whose reply died still re-requests instead of wedging. - Defensive guard in CKnownFilesRem::ProcessUpdate: if the first post-reconnect reply carries no files while the list is still populated, skip the absence-prune and keep the one-shot armed for the next poll. 2. Ellipsis mojibake on Windows The reconnect status strings embedded a literal U+2026. On the untranslated (English) path the narrow msgid is decoded with the C locale (CP1252 on Windows), rendering as garbage in both the dialog and the log. Replaced the ellipsis with "..." in every source msgid, and -- to keep existing translations valid -- in every catalog msgid/msgstr as well (regen leaves no new fuzzy entries). Client-side only: no daemon or EC wire-protocol change, so the updated aMuleGUI works against a stock amuled.
mrjimenez
pushed a commit
to mrjimenez/amule
that referenced
this pull request
Jul 27, 2026
…nnect (amule-project#620) Adding a burst of downloads to a large queue over the remote GUI froze the GUI for seconds and dripped the new files in at only a few per second (issue amule-project#615). Selecting ~100 search results for download on a ~10k queue was the reporter's repro. CKnownFilesRem::ProcessUpdate() only wrapped the list ctrls in BeginBatchUpdate()/EndBatchUpdate() for the post-reconnect reconcile (issue amule-project#444). On an ordinary steady-state poll the batch was never engaged, so each freshly-added partfile went through CDownloadListCtrl::AddFile() with the per-item SortList() active: a full re-sort of the entire list on every insert. For a 10k queue that is O(n^2 log n) and blocks the GUI event loop, which in turn throttles the outbound download requests -- hence the ~4/sec drip the reporter saw. Batch the download list on every non-initial poll: BeginBatchUpdate() suppresses the per-item sort, and the single SortList() runs once at the end, and only when the poll actually added a file (downloadListGrew). A pure in-place stat poll stays sort-free, so the common case pays nothing. EndBatchUpdate() gains a doSort parameter (default true) to express that. The cold-boot m_initialUpdate path keeps its own ShowFileList() batching and is left untouched. The shared-files ctrl batching is unchanged (reconnect-only), matching its existing behaviour. Refs amule-project#615
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Aug 10, 2026
* fix(gui): don't reuse ECIDs across a daemon restart An ECID only means something within one daemon process. CECID hands them out from a counter that restarts with the process, so a restarted amuled reissues the same numbers, in whatever order it loads files that time. amulegui deliberately keeps its objects across a reconnect so scroll and selection survive (amule-project#444), and reconciles the fresh snapshot against them by ECID -- which after a restart pairs each object with whatever now happens to share its number. The row survives and is quietly overwritten with a different file's name, size and statistics, and if the class no longer matches it also ends up in the wrong list. It can also crash. CKnownFilesRem::ProcessItemUpdate copies the part-status array using two bounds from different places: the buffer is as long as the decoder made it from what arrived, while GetPartCount() is the object's own, derived from the size it currently believes the file to be. They agree for as long as an ECID keeps meaning the same file. Paired with a file of a different size the loop reads off the end of the heap allocation, which is a good fit for the 0xc0000005 in issue amule-project#884. The daemon now identifies its process in AUTH_OK via EC_TAG_SESSION_ID, and a reconnect compares it. Same value means the socket dropped but the daemon lived -- a sleeping laptop, a dead tunnel -- so the in-place reconcile is right and scroll and selection still survive. A different value, or none at all because the daemon predates the tag, means nothing keyed by ECID can be trusted, and every such container is dropped and repopulated from the next poll. That fallback is what makes this work against daemons already deployed, which is the case the reporter is in. The teardown goes through RemoteContainer::ResetForNewSession(), which drops items one at a time through the existing RemoveItem/DeleteItem path rather than clearing the indices: that path fires the destroy broadcast that makes clients and list controls drop their raw pointers (amule-project#748, amule-project#755), removes the rows from the views and then deletes. CKnownFilesRem also re-arms the cold-boot path so the repopulate is batched through ShowFileList() instead of sorting once per inserted row (amule-project#414). The part-status copy is bounded and its tail cleared regardless. Two bounds from different sources should never be assumed to agree, and leaving the tail would show the previous file's availability under the new one's name. * fix(gui): report a part-status length mismatch instead of only clamping The clamp on its own is silent, and with the session check in place the only way it can now trigger is if that check failed to notice the daemon changed underneath us. That is worth knowing about: clamping leaves no other trace, and the rest of the update goes on applying the same mismatched tag to the same object, so the visible result would be a row quietly describing the wrong file rather than anything that points at the cause. * fix(gui): warn the user when the remote core sends inconsistent file data A debug line was the wrong level for this. It only appears with EC debug logging enabled, which almost nobody turns on, so in the field it would have reported the problem to nobody -- and the problem is one the user can both see and act on: the clamp keeps the copy in bounds, but the rest of the update goes on applying a mismatched tag to the same object, so what they end up looking at is a row describing the wrong file. Reconnecting clears it, which the message now says. Once per session rather than once per file. The check sits in a loop that covers the whole library on every poll, and the condition it reports is library-wide when it happens at all, so at critical level the per-file form would have put thousands of bold lines in front of the user. Regenerates the po catalogs for this string and for the reconnect notice added earlier on this branch. * fix(gui): drop a null guard that only taught the analyser to doubt m_connect Startup() only runs on a successful connect, and the lines just below take CStatistics(*m_connect) and CStatTreeRem(m_connect) unconditionally, so a null m_connect is already undefined behaviour there. Testing it before reading the session id therefore guarded nothing -- but it did assert that the pointer is optional, and Tier-1 clang-tidy read it exactly that way: having seen the null branch it reported the dereference below as reachable with null (clang-analyzer-core.NonNullParamChecker), failing CI on a line this change never touched. The guard in FinishReconnect() stays. Nothing else in that function dereferences m_connect, so it creates no such branch, and falling back to 0 there means "cannot tell which daemon this is", which selects the safe start-over path rather than papering over anything.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two minimal changes to
UploadClient.cppthat significantly improve upload throughput on fast connections by scaling two previously hardcoded constants with the actual slot data rate.Adaptive payload buffer
Replaces the hardcoded 100 KiB per-client payload buffer with a rate-adaptive value:
The 100 KiB stock buffer drains in ~10ms at 10 MB/s, leaving the socket queue empty while
CreateNextBlockPackage()refills synchronously from disk. A larger buffer gives the main thread more headroom between disk reads.Adaptive packet chunk size
Replaces the hardcoded 10 KiB packet split threshold with a rate-adaptive value:
Applied in both
CreateStandardPackets()andCreatePackedPackets().MAX_PACKET_SIZE(2 MB), compatible with all peersBackward compatibility
UploadClient.cpponly, no throttler or socket layer touchedTest results
Tested on a 1 Gbps dedicated server running aMule in Docker:
Note: peak results were achieved together with #436 (uint16 → uint32 speed limits), which removes the 524 Mbps configuration cap. This PR alone still provides significant improvement on connections below that limit.
Recommended configuration for fast connections
For best results, set an explicit
MaxUploadvalue rather than relying on unlimited mode (MaxUpload=0). In unlimited mode aMule ramps up slots slowly based on observed throughput; an explicit value ensures the full slot count is available immediately.MaxUpload=65534(maximum before the uint16 overflow threshold, ~524 Mbps)MaxUpload=100000or higher for gigabit connectionsSetting
MaxUploadto your actual uplink capacity also gives the slot allocation algorithm accurate data to work with.