Defer per-part hash verification to a dedicated worker thread - #498

Summary

Moves CPartFile::FlushBuffer Phase 3 (per-part MD4 verification) off the main thread onto a dedicated CPartFileHashThread worker, gates it behind a 1 s receive-quiescent window, and skips it entirely once the gaplist closes (because CCompletionTask re-reads the file for the ED2K root + AICH tree and subsumes the per-part work).

The visible effect on a fresh download is that the 30–60 s GUI freeze at file completion (every part dirty, all hashed synchronously on the main thread) is gone. On Windows the freeze had been re-shaped as ~1–4 s sample-rate spikes during active receive bursts; with this PR throughput is steady at the LAN-seeder limit with no spikes or stalls.

What's in this PR

The branch is four logical commits.

1. Add CPartFileHashThread for async per-part hash verification

New src/PartFileHashThread.{h,cpp} modelled on the existing CPartFileWriteThread:

• Single dedicated wxThread with a mutex-protected job queue (HashJob { CPartFile*, partNumber, fromAICHRecoveryDataAvailable })
• wxCondition + sticky m_bWorkPending wake flag to avoid lost signals
• Entry() calls pFile->HashSinglePart(partNumber) synchronously off the main thread, then posts a CPartFileHashResultEvent and decrements pFile->m_pendingHashes
• EndThread() joins on shutdown

Wired up on CamuleApp:

• partFileHashThread member constructed in OnInit after partFileWriteThread
• Torn down at the start of OnExit, before downloadqueue is destroyed, so any in-flight HashSinglePart finishes and the m_pendingHashes counter on each CPartFile drops to zero before ~CPartFile waits on it
• m_pendingHashes (std::atomic<int32>) declared on CPartFile so the worker can decrement it without holding any lock

The worker is constructed and torn down inert — no caller enqueues work yet. Commit 2 wires Phase 3 to use it.

2. PartFile: defer per-part hash verification to the async worker

Replaces the synchronous Phase 3 loop in FlushBuffer (which read the just-written part back through the kernel cache and ran MD4 on the main thread) with an async dispatch:

• WriteToBuffer stamps m_nLastBlockReceivedTick = GetTickCount() so FlushBuffer's new quiescent guard (1 s of no receives) can defer hashing during active receive bursts. The original PR download: port eMule's CPartFileWriteThread to offload disk writes #454 objection that hashing competed with the write thread no longer applies once writes have drained.
• FlushBuffer Phase 3 enqueues each dirty part to CPartFileHashThread::QueueHashCheck and returns; the worker calls HashSinglePart off the main thread and posts a CPartFileHashResultEvent that CamuleApp::OnPartFileHashResult dispatches to CPartFile::OnAsyncHashComplete, which runs the original success / failure / AICH-recovery logic.
• Skip Phase 3 at gaplist completion. CCompletionTask re-reads the whole file for the ED2K root hash and AICH Merkle tree — exactly the same bytes the per-part MD4s would touch. Running both is duplicate work; for a fresh file (every part dirty) it's also the source of the 30–60 s freeze.
• Trailing CompleteFile check accepts both PS_EMPTY and PS_READY (LoadPartFile flips status to PS_READY on resume when any part is already complete) and waits on m_pendingHashes <= 0 so we don't close m_hpartfile while the worker is still reading it. The last OnAsyncHashComplete to drain the counter re-runs the check and fires CompleteFile then.
• Final PB_WRITTEN harvest sweep runs before the trailing check. CPartFileWriteThread::Entry decrements m_iWrites and then sets PB_WRITTEN, so an item Phase 2 left as PB_PENDING can transition to PB_WRITTEN after Phase 2 returns; without this sweep the gap-closing write at file completion would leave m_BufferedData_list non-empty and CompleteFile would not fire.
• OnAsyncHashComplete only calls SavePartFile when the part's outcome changed persistent state (corruption, ICH recovery, or removing from m_corrupted_list). The success path for an already-complete part is in-memory only; saving on every event would re-create the freeze in a different place when many results land in quick succession.

Shutdown safety

~CPartFile waits for m_pendingHashes to drain, calls FlushBuffer (which now early-returns under the m_inDestructor gate before Phase 3 fires), then synchronously sync-hashes any parts still flagged dirty in m_aChangedPart. The shutdown sweep covers the cancel-mid-download case where the quiescent guard never opened: without it, .met would be saved with gaplist marking those parts complete and m_corrupted_list empty, and on next launch they would be implicitly verified ("verified iff IsComplete && !IsCorruptedPart"). Verified-vs-corrupt counts are logged.

3. PartFile/DownloadQueue: drain pending hash work at Process tick rate

Adds CPartFile::HasPendingHashWork() and uses it in two places:

• CPartFile::Process flushes the buffer not only on size/time limits but also whenever there's dirty work waiting (m_aChangedPart entries, or — when the gaplist is closed — un-harvested PB_WRITTEN items in the buffer list). Without this, a quiescent download with leftover dirty parts would wait up to 60 s for the next BUFFER_TIME_LIMIT tick before the async hash queue drained, and at file completion the trailing CompleteFile check would not be re-evaluated until the worker finished the last write.
• CDownloadQueue::Process drives FlushBuffer for paused / insufficient files (their Process() is gated above, so Phase 3 would otherwise never run on dirty pre-pause data). PS_ERROR is intentionally excluded.

4. PartFile: serialise m_hpartfile access against the hash thread

Live testing of commits 1-3 surfaced a corruption-causing race when the user pauses a download (which drains m_aChangedPart through Phase 3 to the hash thread) and then resumes mid-hash: the write thread restarts while the hash thread is still chewing through the queue, and the two race on the underlying file.

Root cause: with ENABLE_MMAP=OFF (the default in the project's CMake), CFileAutoClose::ReadAt and WriteAt both implement positional I/O as Seek(off); Read/Write(...) on the same OS fd. The hash thread's read in HashSinglePart and the write thread's write in FlushAt therefore share the OS file's position; concurrent execution interleaves seeks and lands one operation at the wrong offset.

Fix: add std::mutex CPartFile::m_hpartfileMutex (whole-file lock — one mutex per CPartFile), held around:

• CPartFileWriteThread::Entry's pBuffer->area.FlushAt(...)
• CPartFileHashThread::Entry's HashSinglePart(...)
• ~CPartFile's sync-hash drain HashSinglePart(...)
• FlushBuffer's PB_READY synchronous fallback FlushAt(...) (defensive — the fallback only runs when the write thread is unavailable)

A per-chunk lock would not help: the race is on the shared fd's position, not on the bytes themselves. Two threads reading/writing non-overlapping byte ranges still both call Seek on the same fd. The whole-file lock has small cost — writes are ~1 ms each, hashes ~50 ms each on Windows VM — and the write thread can stall briefly behind a hash backlog, but the main thread is unaffected.

If the serialisation later shows a measurable throughput regression on the LAN-seeder benchmark, the cleanest follow-up is to give the hash thread its own dup()'d fd; with two independent fds and their own positions, they can run truly concurrently. Not needed for correctness; only if measurements demand it.

Benchmarks

LAN seeder → leecher, single source, 90 s ramp benchmark with 1 s polling via amulecmd over external connections. Two metrics matter:

• dl_mbps — instantaneous download throughput
• sample_ms — wall-clock duration of the amulecmd show transfers round-trip; this is the EC reply lag and is the most direct proxy for "is the main thread responsive". A synchronous Phase 3 hash on the main thread blocks every EC reply until it completes, so a per-sample multi-second sample_ms spike maps 1-to-1 onto the GUI freeze the user sees.

macOS (Apple Silicon, Release)

Mac is fast enough (NVMe + Apple Silicon cores) that even a synchronous Phase 3 doesn't bleed into the EC reply lag — no regression, no improvement. Throughput is at the LAN seeder's serving rate on both.

Windows ARM64 (UTM VM, Release)

Windows under UTM is the regime where the synchronous Phase 3 hash shows up clearly: the master baseline has a max EC reply lag of 3.77 s and 2.8% of samples lag past 1 s — every one of those is a multi-second main-thread freeze. With this PR the worst sample is 218 ms (a 17× improvement at max, 10× at p99) and no sample crosses 500 ms. Mean throughput is also up 6.1%.

Backward compatibility

• No wire protocol changes; no .met format changes; no on-disk layout changes.
• Resume from any existing .met works unchanged — the new m_aChangedPart / m_pendingHashes / m_inDestructor / m_nLastBlockReceivedTick state is in-memory only and initialised at construction / load.
• Corrupted-part detection paths (m_corrupted_list, AICH recovery, CCompletionTask post-download rehash, CorruptionBlackBox) are unchanged. Per-part MD4 verification still happens — it just runs on the worker thread now and is skipped when the gaplist closes (because CCompletionTask is about to re-hash everything anyway).
• Pause / resume preserves dirty-part state via ~CPartFile's sync-hash drain at shutdown and m_aChangedPart re-marking on FlushBuffer Phase 2 harvest after resume.
• Synchronous fallback in CPartFile::FlushBuffer Phase 2 (when pThread && pThread->IsRunning() is false) is preserved unchanged.

Edge cases tested

• Fresh download to completion — file completes, AICH hashset is stored, file moves to incoming.
• Resume from partial .met — status loads as PS_READY, downloads resume, gaplist closes, CompleteFile fires (the trailing-check PS_EMPTY || PS_READY acceptance covers this).
• Corruption mid-download — corrupt parts are detected by Phase 3 once quiescent, gaps re-added, AICH recovery requested, peers re-serve.
• Forced quit mid-download — ~CPartFile sync-hashes leftover dirty parts; .met saves with correct gaps and m_corrupted_list; resume re-downloads them.
• Pause / resume mid-download — CDownloadQueue::Process paused-file drain branch runs Phase 3 on leftover dirty parts; AICH recovery still fires.
• Post-completion full-file rehash finds corruption — PartFileHashFinished adds gaps for corrupt parts, file goes back to PS_READY, peers re-serve, the second post-download rehash succeeds. (Pre-existing behaviour; the "Failed to store new AICH Hashset" debug log is the existing wording for "skipped because corruption was found".)

Files changed