Skip to content

upload: adaptive payload buffer and packet chunk size for fast connections - #444

Closed
got3nks wants to merge 2 commits into
amule-project:masterfrom
got3nks:adaptive-upload-buffer
Closed

upload: adaptive payload buffer and packet chunk size for fast connections#444
got3nks wants to merge 2 commits into
amule-project:masterfrom
got3nks:adaptive-upload-buffer

Conversation

@got3nks

@got3nks got3nks commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Two minimal changes to UploadClient.cpp that significantly improve upload throughput on fast connections by scaling two previously hardcoded constants with the actual slot data rate.

Adaptive payload buffer

Replaces the hardcoded 100 KiB per-client payload buffer with a rate-adaptive value:

const uint32 payloadBufferLimit = std::min(std::max(GetUploadDatarate() * 10u, 180u*1024u), 16u*1024u*1024u);
  • Floor: 180 KiB — matches eMule's slow-connection baseline (EMBLOCKSIZE × 1)
  • Scales to ~10 seconds worth of data at the current slot rate
  • Ceiling: 16 MiB — prevents unbounded memory use

The 100 KiB stock buffer drains in ~10ms at 10 MB/s, leaving the socket queue empty while CreateNextBlockPackage() refills synchronously from disk. A larger buffer gives the main thread more headroom between disk reads.

Adaptive packet chunk size

Replaces the hardcoded 10 KiB packet split threshold with a rate-adaptive value:

const uint32 chunkSize = std::min(std::max(GetUploadDatarate() / 8u, 10240u), 131072u);

Applied in both CreateStandardPackets() and CreatePackedPackets().

  • Floor: 10 KiB — preserves stock behaviour on slow connections
  • Ceiling: 128 KiB — well within MAX_PACKET_SIZE (2 MB), compatible with all peers
  • Larger packets reduce per-packet header overhead and throttler loop iterations at high speed

Backward compatibility

  • Connections below ~80 KiB/s use exactly the same constants as stock aMule
  • No wire protocol changes — compatible with all eMule/aMule clients
  • Both changes are in UploadClient.cpp only, no throttler or socket layer touched

Test results

Tested on a 1 Gbps dedicated server running aMule in Docker:

  • Before (stock aMule): ~450 KB/s per upload slot
  • After: 7–8 MB/s per upload slot sustained

Note: peak results were achieved together with #436 (uint16 → uint32 speed limits), which removes the 524 Mbps configuration cap. This PR alone still provides significant improvement on connections below that limit.

Recommended configuration for fast connections

For best results, set an explicit MaxUpload value rather than relying on unlimited mode (MaxUpload=0). In unlimited mode aMule ramps up slots slowly based on observed throughput; an explicit value ensures the full slot count is available immediately.

Setting MaxUpload to your actual uplink capacity also gives the slot allocation algorithm accurate data to work with.

got3nks added 2 commits April 13, 2026 22:16
Buffer ~10 seconds worth of data per upload slot, clamped between
180 KiB (eMule minimum) and 16 MiB (upper bound):

    payloadBufferLimit = clamp(GetUploadDatarate() * 10, 180 KiB, 16 MiB)

Compensates for aMule's synchronous disk I/O: the throttler can drain
the buffer between CreateNextBlockPackage() calls, so a larger lookahead
prevents socket starvation on fast slots. Intended as a temporary measure
until an async disk I/O thread (eMule-style) is implemented.
Scale chunk size with actual slot throughput, floor 10 KiB (stock aMule),
ceil 128 KiB:

    chunkSize = clamp(GetUploadDatarate() / 8, 10 KiB, 128 KiB)

Slow slots keep small chunks for smooth flow; fast slots (>1 MB/s) get
128 KiB chunks to reduce per-packet header overhead and socket queue
churn.
@got3nks
got3nks force-pushed the adaptive-upload-buffer branch from ab6af17 to de3ac51 Compare April 13, 2026 20:16
got3nks added a commit to got3nks/amule that referenced this pull request Apr 15, 2026
Scale upload packet chunk size with the per-slot upload datarate:
  min(max(datarate/8, 10240), 131072)

At low speeds the default 10 KiB chunks keep latency low.  At high speeds
(e.g. LAN or fast peers) chunks grow up to 128 KiB, reducing per-packet
overhead from ed2k framing, encryption, and syscalls.

Benchmark on LAN (Docker, 64 MB/s upload cap):
  Before: 20 MB/s avg, 61706 chunks in 30s
  After:  59 MB/s avg, 13051 chunks in 30s  (3x throughput)

ref: amule-project#444
@got3nks

got3nks commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #451, which adopts eMule's CUploadDiskIOThread approach end-to-end (background disk I/O + packet construction) and keeps the adaptive chunk sizing from this PR on top of it. Closing in favour of that.

@got3nks got3nks closed this Apr 16, 2026
@got3nks
got3nks deleted the adaptive-upload-buffer branch May 3, 2026 15:19
ngosang pushed a commit to ngosang/amule that referenced this pull request Jul 12, 2026
amule-project#444) (amule-project#452)

When the EC connection to the remote core dropped after startup (e.g. the
machine slept or a VPN/SSH tunnel restarted and the socket died), amulegui
showed "aMule has terminated probably" and exited, forcing a full restart +
resync. It now reconnects in the background instead.

Reconnect mechanics:
- CLibSocket::ResetForReconnect() swaps a fresh asio impl onto the SAME
  CRemoteConnect, so the socket can be re-opened after a loss without
  recreating the object (every remote container pins its CRemoteConnect).
  CECSocket::ResetProtocolState() rewinds the packet-reassembly state
  machine so a mid-packet read left over from the drop can't misparse the
  reconnected session's first bytes.
- On a post-startup loss the UI is frozen behind a modal reconnect dialog
  (attempt counter + countdown to the next try + "Abort and exit"); attempts
  run every 5 s with a 15 s per-attempt watchdog, until the connection is
  restored or the user aborts.
- amuleweb / amulecmd keep their fail-fast _exit(1) on loss (the new paths
  are all gated on the GUI's m_notifier).

Two fixes the reused socket needs to hand off cleanly:
- Clear m_ErrorCode on a successful connect: the swap can leave a stale
  EBADF from an aborted read on the impl, and SocketRealError() would then
  make CECSocket::WritePacket refuse to send the login.
- Swallow a LibSocketLost that arrives while a dial is still pending
  (EC_CONNECT_SENT): it's a stale event queued for the previous connection
  before the impl was swapped, and would otherwise abort the in-flight
  reconnect. A genuinely failed dial is caught by the connect-timeout
  watchdog instead.

Resync is reconcile-in-place, not wipe+rebuild, so scroll and selection
survive:
- Polling resumes and the fresh full snapshot updates rows in place / adds
  new / prunes gone. A partial-update server won't re-emit FILE_REMOVED for
  files deleted while disconnected, so CKnownFilesRem forces a one-shot
  prune-by-absence on the first post-reconnect update (m_reconnectReconcile).
  The server / client / friend lists already prune every poll, so they
  self-heal with no special case.
- The daemon keeps its RLE gap/part/req-status encoders per connection, so a
  reconnect restarts them from an empty baseline. The reused CKnownFile /
  CPartFile decoders (m_PartFileEncoderData, m_partStatus) are reset to
  match, or the first differential update would XOR against a stale buffer
  and paint garbage (all-red progress bars, wrong availability shading).
- Large libraries stay responsive: the reconnect poll wraps the download +
  shared list ctrls in BeginBatchUpdate()/EndBatchUpdate() (one repaint +
  one sort, per-item sort suppressed) so a 10k+ resync doesn't hitch.

No daemon or EC wire-protocol changes: reconnect works against a stock amuled.
ngosang pushed a commit to ngosang/amule that referenced this pull request Jul 13, 2026
amule-project#444) (amule-project#465)

Follow-up to amule-project#452, from two issues reported after a successful reconnect.

1. Empty download queue + shared file lists (restart required)

   The EC request FIFO (CRemoteConnect::m_req_fifo) assumes the core answers
   every request in FCFS order. A socket dropped mid-poll leaves the requests
   that were on the air unanswered, so their handlers linger in the FIFO. After
   the reconnect each reply pops the wrong (stale) handler: a stats reply routed
   to CKnownFilesRem drives its one-shot post-reconnect reconcile against an
   empty file set, and the absence-prune wipes the whole library. It is
   intermittent (depends how many requests were in flight at the drop) and only
   a full restart clears it, since a fresh process starts with an empty FIFO.

   - CRemoteConnect::DiscardRequestQueue() flushes the FIFO and zeroes the
     in-flight counter on reconnect, rewinding each orphaned handler's request
     state (CECPacketHandlerBase::AbortPendingRequest, overridden by
     CRemoteContainer to reset its request SM to IDLE) so a container whose
     reply died still re-requests instead of wedging.
   - Defensive guard in CKnownFilesRem::ProcessUpdate: if the first
     post-reconnect reply carries no files while the list is still populated,
     skip the absence-prune and keep the one-shot armed for the next poll.

2. Ellipsis mojibake on Windows

   The reconnect status strings embedded a literal U+2026. On the untranslated
   (English) path the narrow msgid is decoded with the C locale (CP1252 on
   Windows), rendering as garbage in both the dialog and the log. Replaced the
   ellipsis with "..." in every source msgid, and -- to keep existing
   translations valid -- in every catalog msgid/msgstr as well (regen leaves no
   new fuzzy entries).

Client-side only: no daemon or EC wire-protocol change, so the updated aMuleGUI
works against a stock amuled.
mrjimenez pushed a commit to mrjimenez/amule that referenced this pull request Jul 27, 2026
…nnect (amule-project#620)

Adding a burst of downloads to a large queue over the remote GUI froze the
GUI for seconds and dripped the new files in at only a few per second
(issue amule-project#615). Selecting ~100 search results for download on a ~10k queue
was the reporter's repro.

CKnownFilesRem::ProcessUpdate() only wrapped the list ctrls in
BeginBatchUpdate()/EndBatchUpdate() for the post-reconnect reconcile
(issue amule-project#444). On an ordinary steady-state poll the batch was never
engaged, so each freshly-added partfile went through
CDownloadListCtrl::AddFile() with the per-item SortList() active: a full
re-sort of the entire list on every insert. For a 10k queue that is
O(n^2 log n) and blocks the GUI event loop, which in turn throttles the
outbound download requests -- hence the ~4/sec drip the reporter saw.

Batch the download list on every non-initial poll: BeginBatchUpdate()
suppresses the per-item sort, and the single SortList() runs once at the
end, and only when the poll actually added a file (downloadListGrew). A
pure in-place stat poll stays sort-free, so the common case pays nothing.
EndBatchUpdate() gains a doSort parameter (default true) to express that.
The cold-boot m_initialUpdate path keeps its own ShowFileList() batching
and is left untouched. The shared-files ctrl batching is unchanged
(reconnect-only), matching its existing behaviour.

Refs amule-project#615
got3nks added a commit to got3nks/amule that referenced this pull request Aug 10, 2026
* fix(gui): don't reuse ECIDs across a daemon restart

An ECID only means something within one daemon process. CECID hands them
out from a counter that restarts with the process, so a restarted amuled
reissues the same numbers, in whatever order it loads files that time.
amulegui deliberately keeps its objects across a reconnect so scroll and
selection survive (amule-project#444), and reconciles the fresh snapshot against them by
ECID -- which after a restart pairs each object with whatever now happens to
share its number. The row survives and is quietly overwritten with a
different file's name, size and statistics, and if the class no longer
matches it also ends up in the wrong list.

It can also crash. CKnownFilesRem::ProcessItemUpdate copies the part-status
array using two bounds from different places: the buffer is as long as the
decoder made it from what arrived, while GetPartCount() is the object's own,
derived from the size it currently believes the file to be. They agree for
as long as an ECID keeps meaning the same file. Paired with a file of a
different size the loop reads off the end of the heap allocation, which is a
good fit for the 0xc0000005 in issue amule-project#884.

The daemon now identifies its process in AUTH_OK via EC_TAG_SESSION_ID, and
a reconnect compares it. Same value means the socket dropped but the daemon
lived -- a sleeping laptop, a dead tunnel -- so the in-place reconcile is
right and scroll and selection still survive. A different value, or none at
all because the daemon predates the tag, means nothing keyed by ECID can be
trusted, and every such container is dropped and repopulated from the next
poll. That fallback is what makes this work against daemons already
deployed, which is the case the reporter is in.

The teardown goes through RemoteContainer::ResetForNewSession(), which drops
items one at a time through the existing RemoveItem/DeleteItem path rather
than clearing the indices: that path fires the destroy broadcast that makes
clients and list controls drop their raw pointers (amule-project#748, amule-project#755), removes the
rows from the views and then deletes. CKnownFilesRem also re-arms the
cold-boot path so the repopulate is batched through ShowFileList() instead
of sorting once per inserted row (amule-project#414).

The part-status copy is bounded and its tail cleared regardless. Two bounds
from different sources should never be assumed to agree, and leaving the
tail would show the previous file's availability under the new one's name.

* fix(gui): report a part-status length mismatch instead of only clamping

The clamp on its own is silent, and with the session check in place the only
way it can now trigger is if that check failed to notice the daemon changed
underneath us. That is worth knowing about: clamping leaves no other trace,
and the rest of the update goes on applying the same mismatched tag to the
same object, so the visible result would be a row quietly describing the
wrong file rather than anything that points at the cause.

* fix(gui): warn the user when the remote core sends inconsistent file data

A debug line was the wrong level for this. It only appears with EC debug
logging enabled, which almost nobody turns on, so in the field it would have
reported the problem to nobody -- and the problem is one the user can both
see and act on: the clamp keeps the copy in bounds, but the rest of the
update goes on applying a mismatched tag to the same object, so what they
end up looking at is a row describing the wrong file. Reconnecting clears
it, which the message now says.

Once per session rather than once per file. The check sits in a loop that
covers the whole library on every poll, and the condition it reports is
library-wide when it happens at all, so at critical level the per-file form
would have put thousands of bold lines in front of the user.

Regenerates the po catalogs for this string and for the reconnect notice
added earlier on this branch.

* fix(gui): drop a null guard that only taught the analyser to doubt m_connect

Startup() only runs on a successful connect, and the lines just below take
CStatistics(*m_connect) and CStatTreeRem(m_connect) unconditionally, so a
null m_connect is already undefined behaviour there. Testing it before
reading the session id therefore guarded nothing -- but it did assert that
the pointer is optional, and Tier-1 clang-tidy read it exactly that way:
having seen the null branch it reported the dereference below as reachable
with null (clang-analyzer-core.NonNullParamChecker), failing CI on a line
this change never touched.

The guard in FinishReconnect() stays. Nothing else in that function
dereferences m_connect, so it creates no such branch, and falling back to 0
there means "cannot tell which daemon this is", which selects the safe
start-over path rather than papering over anything.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant