Skip to content

Stream large file copies through a 1 MiB buffer (#11) - #266

Merged
got3nks merged 1 commit into
amule-org:masterfrom
got3nks:fix/slow-cross-fs-copy
Jul 1, 2026
Merged

Stream large file copies through a 1 MiB buffer (#11)#266
got3nks merged 1 commit into
amule-org:masterfrom
got3nks:fix/slow-cross-fs-copy

Conversation

@got3nks

@got3nks got3nks commented Jul 1, 2026

Copy link
Copy Markdown

Problem

wxCopyFile streams through a hard-coded 4 KiB buffer. On download completion aMule moves the finished file from Temp to Incoming; when those directories live on different filesystems the rename fails and it falls back to a copy — and that copy runs through the 4 KiB buffer. On network filesystems (NFS, sshfs) the tiny buffer defeats write coalescing and pays a round-trip per block, throttling completion to a fraction of line speed. Reported in #11 at ~13 MB/s on NFSv3.

Fix

Add CFile::CloneFile, which streams through a 1 MiB heap buffer, and route the large data-file copies through it: the completion move (ThreadTasks) and the eMule part-file import (PartFileConvert), both of which can cross filesystems.

The helper lives on CFile rather than CPath for a layering reason: CPath is in the low-level mulecommon library, which must not depend on the CFile layer, and all four former CPath::CloneFile callers already sit in the CFile layer. CPath::CloneFile is therefore removed. Its two small, same-filesystem users stay on the plain wxCopyFile path: CPath::BackupFile inlines it, and the clients.met backup in ClientCreditsList switches to CPath::BackupFile.

The implementation is plain CFile read/write with no platform #ifdefs, so it behaves uniformly across macOS, Windows, Linux and BSD. The buffer is heap-allocated (not a stack array) because the copy can run on the completion worker thread, where musl caps the thread stack at 128 KiB. On any error the partial destination is removed, so a failed completion never leaves a corrupt file behind.

Benchmark

Cold copy of a 2 GB file across an sshfs-to-localhost mount (cache dropped between runs), plus read/write syscall counts for a 256 MB copy:

Buffer Cross-filesystem throughput read/write syscalls (256 MB)
4 KiB (old wxCopyFile) 16.7 MB/s 131,075
1 MiB (new CFile::CloneFile) 344.1 MB/s 515

~20× faster on the cross-filesystem path (the 4 KiB figure matches the ~13 MB/s from the report), with 256× fewer syscalls.

Testing

New CloneFileTest covers byte-identical multi-chunk copies, empty files, overwrite semantics, the failure paths (missing source, unwritable destination — no partial file left behind), and CPath::BackupFile (the small same-filesystem path the clients.met backup now uses). It also carries an env-gated case, AMULE_CLONEFILE_HUGE_GB, that copies a real >4 GiB file to exercise the 64-bit offset path; I ran it with a 5 GiB file (length + streaming content verified). Full unit suite green; clang-format clean.

Closes #11.

wxCopyFile streams through a hard-coded 4 KiB buffer, which throttled the
Temp -> Incoming copy on download completion -- taken when the two
directories live on different filesystems -- to a fraction of line speed
on NFS / sshfs.

Move the large data-file copies to CFile::CloneFile, which streams through
a 1 MiB heap buffer: the completion move (ThreadTasks) and the eMule
part-file import (PartFileConvert), both of which can cross filesystems.
CFile lives in the src/ layer where those callers already are.

CPath::CloneFile (in mulecommon, which must not depend on the CFile layer)
is removed. Its two small same-filesystem users stay on the plain
wxCopyFile path: CPath::BackupFile inlines it, and the clients.met backup
in ClientCreditsList switches to CPath::BackupFile.

Add CloneFileTest: byte-identical multi-chunk copy, empty file, overwrite
semantics, missing-source and unwritable-destination paths (no partial
file left behind), and an env-gated >4 GiB case (AMULE_CLONEFILE_HUGE_GB)
covering the 64-bit offset path.
@got3nks
got3nks merged commit f972d7b into amule-org:master Jul 1, 2026
11 checks passed
@got3nks
got3nks deleted the fix/slow-cross-fs-copy branch July 1, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

File completion: Slow copy speed because of wxwidgets restriction

1 participant