Stream large file copies through a 1 MiB buffer (#11) - #266
Merged
Conversation
wxCopyFile streams through a hard-coded 4 KiB buffer, which throttled the Temp -> Incoming copy on download completion -- taken when the two directories live on different filesystems -- to a fraction of line speed on NFS / sshfs. Move the large data-file copies to CFile::CloneFile, which streams through a 1 MiB heap buffer: the completion move (ThreadTasks) and the eMule part-file import (PartFileConvert), both of which can cross filesystems. CFile lives in the src/ layer where those callers already are. CPath::CloneFile (in mulecommon, which must not depend on the CFile layer) is removed. Its two small same-filesystem users stay on the plain wxCopyFile path: CPath::BackupFile inlines it, and the clients.met backup in ClientCreditsList switches to CPath::BackupFile. Add CloneFileTest: byte-identical multi-chunk copy, empty file, overwrite semantics, missing-source and unwritable-destination paths (no partial file left behind), and an env-gated >4 GiB case (AMULE_CLONEFILE_HUGE_GB) covering the 64-bit offset path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
wxCopyFilestreams through a hard-coded 4 KiB buffer. On download completion aMule moves the finished file from Temp to Incoming; when those directories live on different filesystems the rename fails and it falls back to a copy — and that copy runs through the 4 KiB buffer. On network filesystems (NFS, sshfs) the tiny buffer defeats write coalescing and pays a round-trip per block, throttling completion to a fraction of line speed. Reported in #11 at ~13 MB/s on NFSv3.Fix
Add
CFile::CloneFile, which streams through a 1 MiB heap buffer, and route the large data-file copies through it: the completion move (ThreadTasks) and the eMule part-file import (PartFileConvert), both of which can cross filesystems.The helper lives on
CFilerather thanCPathfor a layering reason:CPathis in the low-levelmulecommonlibrary, which must not depend on theCFilelayer, and all four formerCPath::CloneFilecallers already sit in theCFilelayer.CPath::CloneFileis therefore removed. Its two small, same-filesystem users stay on the plainwxCopyFilepath:CPath::BackupFileinlines it, and theclients.metbackup inClientCreditsListswitches toCPath::BackupFile.The implementation is plain
CFileread/write with no platform#ifdefs, so it behaves uniformly across macOS, Windows, Linux and BSD. The buffer is heap-allocated (not a stack array) because the copy can run on the completion worker thread, where musl caps the thread stack at 128 KiB. On any error the partial destination is removed, so a failed completion never leaves a corrupt file behind.Benchmark
Cold copy of a 2 GB file across an sshfs-to-localhost mount (cache dropped between runs), plus read/write syscall counts for a 256 MB copy:
wxCopyFile)CFile::CloneFile)~20× faster on the cross-filesystem path (the 4 KiB figure matches the ~13 MB/s from the report), with 256× fewer syscalls.
Testing
New
CloneFileTestcovers byte-identical multi-chunk copies, empty files, overwrite semantics, the failure paths (missing source, unwritable destination — no partial file left behind), andCPath::BackupFile(the small same-filesystem path theclients.metbackup now uses). It also carries an env-gated case,AMULE_CLONEFILE_HUGE_GB, that copies a real >4 GiB file to exercise the 64-bit offset path; I ran it with a 5 GiB file (length + streaming content verified). Full unit suite green; clang-format clean.Closes #11.