download: port eMule's CPartFileWriteThread to offload disk writes by got3nks · Pull Request #454 · amule-project/amule

got3nks · 2026-04-19T18:25:03Z

Summary

Offloads download disk writes from the main thread to a dedicated write thread (ported from eMule's CPartFileWriteThread), eliminating FlushBuffer stalls that block the main event loop and throttle download throughput.

What's in this PR

1. Write thread

New CPartFileWriteThread (src/PartFileWriteThread.{h,cpp}) owns all disk writes for download buffers. The main thread no longer blocks on file I/O during FlushBuffer() — it queues items to the write thread and harvests completed writes on the next call.

Design choices vs eMule:

wxThread + wxCondition replaces CWinThread + IOCP / GetQueuedCompletionStatus
Synchronous CFileArea::FlushAt() replaces WriteFile(OVERLAPPED) (no need for completion port; writes are serialised on this thread)
Sticky wake flag (m_bWorkPending) prevents lost signals from wxCondition's pulse semantics

FlushBuffer() rewritten with a two-phase approach:

Phase 1: queue PB_READY items to the write thread (set PB_PENDING, increment m_iWrites)
Phase 2: harvest PB_WRITTEN items, track changed parts via m_aChangedPart, free memory
Synchronous fallback when write thread is not running (remote GUI, shutdown)
Hash verification deferred until m_iWrites reaches zero — prevents hashing data not yet on disk
CompleteFile guard: only when m_gaplist.IsComplete() && m_iWrites <= 0 && m_BufferedData_list.empty()

Converges several behaviours with eMule that aMule previously lacked:

PartFileBufferedData moved from .cpp to .h with flushed state field (eMule ref: PB_READY/PB_PENDING/PB_ERROR/PB_WRITTEN)
m_aChangedPart made persistent across FlushBuffer() calls (was local std::vector<bool> recreated each call)
m_iWrites counter tracks in-flight write items, used as guard for hash verification and file completion

2. `m_iWrites` thread safety

m_iWrites is shared between the main thread (++m_iWrites in FlushBuffer) and the write thread (--m_iWrites after each write). eMule uses InterlockedDecrement on Windows; the aMule port uses std::atomic<int32> for the same guarantee. Without atomics, non-atomic read-modify-write on ARM (Apple M-series) causes lost updates — m_iWrites goes negative, bypassing the hash-verification guard.

3. Remote GUI build

PartFileWriteThread.cpp added to CORE_SOURCES (not COMMON_SOURCES) in both cmake and Makefile.am. The remote GUI (amulegui) does not have m_hpartfile and cannot compile the write thread — placing it in CORE_SOURCES ensures it is only built for amuled and the monolithic GUI.

Benchmarks

Tested with a direct LAN transfer between two aMule instances sharing a 30 GB test file. The server (uploader) runs all submitted PRs (#436, #451, #452, #453).

Setup:

Server (uploader): bare metal x86-64 (Ubuntu), 10 Gbps NIC, MaxUpload=300000 KB/s, SlotAllocation=50 KB/s
Client (downloader): Mac Studio M2, 2.5 Gbps NIC, same LAN switch
File: 30 GB random data
Measurement: TCP bytes_sent sampled from ss -ti every 10 seconds (zero instrumentation in the code path)

Client on master + server with #436, #451, #452, #453

Sustained: ~31 MB/s
Peak:       35 MB/s

Client with this PR (write thread) + server with #436, #451, #452, #453

Sustained: ~72 MB/s
Peak:       73 MB/s

2.3x download throughput improvement with no decay over 3+ minutes. The main thread remains responsive throughout — disk writes are fully offloaded to the background thread.

Also tested with amule (monolithic GUI app)

Backward compatibility

No wire protocol changes — works with all stock eMule / aMule clients.
FlushBuffer() falls back to synchronous writes when the write thread is not running — remote GUI and shutdown paths are unchanged.
Hash verification behaviour is preserved — same parts are checked, just deferred until writes complete.

Trade-offs

Hash verification is deferred during active download:

Normal path: m_iWrites > 0 while the write thread has pending items. At sustained high speeds (60+ MB/s) m_iWrites rarely drops to 0 until the download pauses or completes — so HashSinglePart() does not run mid-download. On pause or completion, all accumulated m_aChangedPart parts are hashed in one pass on the main thread.
Clean shutdown: EndThread() drains the write queue before CompleteFile() — the .met is saved with hash-verified state.
Crash / hard restart during active download: the .met may be out of date relative to the .part file. On startup aMule detects the timestamp mismatch and triggers CHashingTask (existing mechanism), which rehashes the entire file on a background thread and restores state via PartFileHashFinished() → "Found completed part" log lines. Startup is not blocked (hashing runs in the background).

An alternative design using a dedicated hash thread for incremental per-part verification was prototyped (branch kept locally for future work). It eliminated the post-crash rehash but added measurable throughput cost:

amuled daemon: ~72 MB/s → ~65 MB/s (~10% overhead)
Monolithic GUI: ~65 MB/s → ~30 MB/s (~55% overhead)

The GUI's main thread is already busy servicing wxWidgets events (list control redraws, progress bar updates, etc.); adding per-part hash result dispatch on top pushed it over. The cost of one full rehash after a crash was deemed preferable to halving sustained GUI throughput.

Files changed

File	Purpose
`src/PartFileWriteThread.h`	New write thread header
`src/PartFileWriteThread.cpp`	New write thread implementation
`src/PartFile.cpp`	Two-phase FlushBuffer; deferred hash verification
`src/PartFile.h`	`PartFileBufferedData` moved here; `m_iWrites` (atomic), `m_aChangedPart`
`src/amule.cpp` / `.h`	Start / stop the write thread
`src/Makefile.am`	Build `PartFileWriteThread.cpp` (CORE_SOURCES)
`cmake/source-vars.cmake`	Build `PartFileWriteThread.cpp` (cmake, CORE_SOURCES)

Offload download disk writes from the main thread to a dedicated background thread, ported from eMule's CPartFileWriteThread. The main thread no longer blocks on CFileArea::FlushAt() during FlushBuffer(), eliminating stalls that froze all socket I/O, upload service, and UI updates while writing downloaded data to disk. Design (same pattern as the upload CUploadDiskIOThread port): - wxThread + wxCondition replaces CWinThread + IOCP - Synchronous CFileArea::FlushAt() on the write thread replaces Windows overlapped WriteFile — the thread is dedicated, so blocking on disk I/O doesn't stall any other work - Sticky wake flag prevents lost wxCondition signals FlushBuffer rewritten as two-phase (eMule ref: PartFile.cpp:4102): Phase 1: queue PB_READY items to write thread (set PB_PENDING) Phase 2: harvest PB_WRITTEN items, track changed parts, free memory Synchronous fallback: if write thread not running, write inline PartFileBufferedData gains a flushed state machine (eMule ref): PB_READY -> PB_PENDING -> PB_WRITTEN (or PB_ERROR -> PB_READY retry) Hash verification deferred until all writes complete (m_iWrites <= 0) to avoid reading back data that hasn't hit disk yet. Changed parts tracked persistently in m_aChangedPart (eMule ref: m_aChangedPart) so they survive across FlushBuffer() calls. m_iWrites decremented by the write thread (matching eMule's WriteCompletionRoutine) so the main thread can check completion at any time without harvesting first. File completion guarded: CompleteFile only called when m_gaplist.IsComplete() && m_iWrites <= 0 && m_BufferedData_list.empty() Tested: 43 MB/s sustained download on macOS ARM from LAN peer.

Move PartFileWriteThread.cpp from COMMON_SOURCES to CORE_SOURCES so it is not compiled for amulegui (which lacks m_hpartfile). Make m_iWrites std::atomic<int32> to fix a data race between the main thread (++m_iWrites in FlushBuffer) and the write thread (--m_iWrites after disk write). On ARM (Apple M-series), the non-atomic read-modify-write caused lost updates, making m_iWrites go negative and bypassing the hash-verification guard.

The in-flight-write guard introduced in amule-project#454 (`m_iWrites > 0`) was designed to keep `HashSinglePart()` from running mid-download. In practice it does on a fast LAN at sustained 60+ MB/s where the write thread's queue stays non-empty across TCP receive bursts -- but at 40 MB/s on a Windows VM, individual writes complete fast enough that the queue empties cleanly in the 30-100 ms gaps the kernel leaves between bursts. (Per-WriteFile cost on Windows under UTM virtualisation -- ntdll -> NTFS filter chain -> VM exit on M-series hosts -- still leaves the write thread fast enough to drain the queue at this rate; what kept the queue non-empty in the original PR's tests was higher wire speed, not lower per-write latency.) CPartFile::Process fires FlushBuffer every 100 ms; if it catches one of those gaps with `m_iWrites == 0`, the synchronous Phase-3 per-part hash loop runs and reads back ~9.28 MB per dirty part to MD4-verify. When multiple parts have completed since the last hashable moment, the loop processes them all in one go on the GUI thread. Verified via Process Monitor 2026-04-28: a 3.5 s `sample_ms` spike during sustained download corresponded to 36 ReadFile events on the .part file, each exactly PARTSIZE (9,728,000 bytes) at random offsets. Aggregate 334 MB of synchronous read + MD4 work on the main thread. A short-circuit probe (HashSinglePart returning true unconditionally) eliminated the spikes entirely and lifted sustained throughput from 37 to 41 MB/s on the same workload, confirming hash on the GUI thread as the single dominant cause of the periodic freezes the user has been seeing on Windows. Surgical fix: keep all the existing hashing behaviour, but add a 1-second quiescent-window guard alongside the m_iWrites check. Only run the hash loop when both: - all queued writes have completed (m_iWrites == 0), and - no new block has been delivered to WriteToBuffer in the last kHashQuiescentMs ms (1000 ms). A genuine pause / completion holds the gate open and the next 100 ms FlushBuffer tick runs the accumulated hashes. An inter-burst gap shorter than 1 s never opens it. Per-part hash semantics are preserved -- corruption detection still happens, just deferred from "between TCP bursts" to "after the file goes still." Two small reshapes around the guard: 1. WriteToBuffer now stamps m_nLastBlockReceivedTick on every block arrival. Cheap (one assignment in the hot per-block path). 2. FlushBuffer no longer returns early when m_BufferedData_list is empty. Phases 1 and 2 are no-ops on an empty list, but Phase 3 needs to run when writes have fully drained -- otherwise dirty m_aChangedPart entries from the last flush would never be picked up after the buffer empties, and stale "complete but unverified" parts could persist forever. CheckFreeDiskSpace is also gated on a non-empty buffer (no point checking when no write would happen). Together: the trigger cannot get lost; FlushBuffer keeps ticking and reaches Phase 3 every 100 ms while either work is queued or a paused download has dirty parts to verify.

The in-flight-write guard introduced in amule-project#454 (`m_iWrites > 0`) was designed to keep `HashSinglePart()` from running mid-download. In practice it does on a fast LAN at sustained 60+ MB/s where the write thread's queue stays non-empty across TCP receive bursts -- but at lower wire speeds the queue can empty in the 30-100 ms gaps the kernel leaves between bursts. CPartFile::Process fires FlushBuffer every 100 ms; if it catches one of those gaps with `m_iWrites == 0`, the synchronous Phase-3 per-part hash loop runs and reads back ~PARTSIZE per dirty part to MD4-verify. When several parts have completed since the last hashable moment, the loop processes them all in one go on the GUI thread and freezes it for 1-3 seconds. Surgical fix: add a 1-second quiescent-window guard alongside the m_iWrites check. Only run the hash loop when both: - all queued writes have completed (m_iWrites == 0), and - no new block has been delivered to WriteToBuffer in the last kHashQuiescentMs ms (1000 ms). A genuine pause / completion holds the gate open and the next 100 ms FlushBuffer tick runs the accumulated hashes. An inter-burst gap shorter than 1 s never opens it. Per-part hash semantics are preserved -- corruption detection still happens, just deferred from "between TCP bursts" to "after the file goes still." Two small reshapes around the guard: 1. WriteToBuffer now stamps m_nLastBlockReceivedTick on every block arrival. Cheap (one assignment in the hot per-block path). 2. FlushBuffer no longer returns early when m_BufferedData_list is empty. Phases 1 and 2 are no-ops on an empty list, but Phase 3 needs to run when writes have fully drained -- otherwise dirty m_aChangedPart entries from the last flush would never be picked up after the buffer empties, and stale "complete but unverified" parts could persist forever. CheckFreeDiskSpace is also gated on a non-empty buffer (no point checking when no write would happen). 3. ~CPartFile sets m_inDestructor=true before its FlushBuffer call, and Phase 3 bails when that flag is set or when theApp is shutting down. Without this gate the hash success branch reaches into theApp->sharedfiles->SafeAddKFile(this), which is unsafe during destruction (sharedfiles may already be torn down on shutdown; a delete-file flow shouldn't re-share what's being deleted). Together: the trigger cannot get lost; FlushBuffer keeps ticking and reaches Phase 3 every 100 ms while either work is queued or a paused download has dirty parts to verify, but never during destruction or shutdown.

The 50 ms time-bound on the synchronous Phase 3 hash loop kept the main thread responsive on NVMe (where ~21 ms/HashSinglePart fits the budget at ~2 parts/call), but on slower disks where a single HashSinglePart already exceeds the budget the loop still ran one part per call — and on a 5400 rpm HDD that's a 100-200 ms freeze per call, above CORE_TIMER_PERIOD. Result: same back-to-back OnCoreTimer starvation we just fixed. Move the hash work entirely off the main thread. - New CPartFileHashThread (dedicated wxThread), structurally mirroring CPartFileWriteThread from PR amule-project#454. Pops HashJobs from a mutex-protected queue, calls HashSinglePart on each (pure read+MD4, no shared mutable state), posts a CPartFileHashResultEvent to CamuleApp's event loop with the file's CMD4Hash + part number + result. - CamuleApp::OnPartFileHashResult looks up the file in the download queue by hash (handles "file deleted between enqueue and dispatch") and calls CPartFile::OnAsyncHashComplete, which runs the original Phase 3 success/failure logic — AICH-recovery on MD4 mismatch, SafeAddKFile on a successfully completed part, ICH recovery branch — all on the main thread, all microsecond-fast. - FlushBuffer's Phase 3 inline hash loop is replaced by a queue-and-clear loop: each dirty part increments m_pendingHashes (atomic) and is enqueued. Main thread cost in Phase 3: O(partCount) of cheap iteration, no I/O, no hashing. GUI freeze: gone, on any disk. - ~CPartFile waits for m_pendingHashes to reach 0 before closing m_hpartfile, so the worker is never reading a torn- down file descriptor. Combined with m_inDestructor (which blocks Phase 3 from enqueueing more) that closes the lifetime race. Why this is safe vs PR amule-project#454's rejection of an async-hash thread: PR amule-project#454 worried about the worker's read-for-hash competing with CPartFileWriteThread's write for the same disk during active download. The quiescent guard added earlier in this branch (1 s of no receives before Phase 3 enqueues) means writes have already drained by the time we enqueue, so the contention can't arise. Also: the bypass we added in CPartFile::Process and CDownloadQueue::Process for paused/insufficient files keeps firing FlushBuffer at Process()-tick rate while drain work is pending, so the worker sees a steady stream of HashJobs rather than waiting on the 60 s BUFFER_TIME_LIMIT. Includes AddDebugLogLineN traces at queue insert / pop / hash complete / event drop for testing visibility (logPartFile + VerboseDebug=1).

Replaces the synchronous Phase 3 loop in FlushBuffer (which read the just-written part back through the kernel cache and ran MD4 on the main thread, freezing the GUI for 30-60 s on a fresh download where every part is dirty) with an async dispatch: * WriteToBuffer stamps m_nLastBlockReceivedTick so FlushBuffer's new quiescent guard (1 s of no receives) can defer hashing during active receive bursts; the original PR-#454 objection that hashing competed with the write thread no longer applies once writes have drained. * FlushBuffer enqueues each dirty part to CPartFileHashThread and returns; the worker calls HashSinglePart off the main thread and posts a CPartFileHashResultEvent that CamuleApp::OnPartFileHashResult dispatches to CPartFile::OnAsyncHashComplete, which runs the original success / failure / AICH-recovery logic. * m_pendingHashes counts in-flight HashJobs targeting the file so CompleteFile and ~CPartFile know when it is safe to close m_hpartfile. * Phase 3 is skipped at gaplist completion: CCompletionTask re-reads the file for the ED2K root + AICH tree and subsumes per-part MD4 — running both is duplicate work and (for a fresh file) the source of the freeze. * The trailing CompleteFile check accepts both PS_EMPTY and PS_READY (LoadPartFile flips status to PS_READY on resume when any part is already complete) and waits on m_pendingHashes <= 0 so we do not close m_hpartfile while the worker is still reading it. * A final PB_WRITTEN harvest sweep runs before the trailing check — CPartFileWriteThread decrements m_iWrites and then sets PB_WRITTEN, so an item Phase 2 left as PB_PENDING can transition to PB_WRITTEN after Phase 2 returns; without this sweep the gap-closing write would leave m_BufferedData_list non-empty and CompleteFile would not fire. * OnAsyncHashComplete only calls SavePartFile when the part's outcome changed persistent state — the success path for an already-complete part is in-memory only, and saving on every event would re-create the freeze in a different place. ~CPartFile waits for m_pendingHashes to drain, calls FlushBuffer (which now early-returns under the m_inDestructor gate before Phase 3 fires), then synchronously sync-hashes any parts still flagged dirty in m_aChangedPart. The shutdown sweep covers the cancel-mid-download case where the quiescent guard never opened: without it, .met would be saved with gaplist marking those parts complete and m_corrupted_list empty, and on next launch they would be implicitly verified.

Two tables at the top of the Performance section in 3.0.0: 1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over LAN, 30 GB file: macOS: 0.35 -> 135 MB/s (381x) Linux: 0.34 -> 117 MB/s (345x) Windows: 0.36 -> 39 MB/s (107x) Plus a per-platform seeder/leecher contribution split that shows the seeder-side fix (amule-project#451) dominates and the leecher-side stack (amule-project#454/amule-project#484/amule-project#491) adds another 4.5-5.8x on top. 2. vs eMule 0.70b on Windows — same UTM hardware, both directions: Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x) Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x) Methodology footnote on the table since eMule has no remote- control protocol and is measured leecher-side via aMule's EC channel on the receiving box. Highlights line 16 also updated to mention both directions of the eMule comparison instead of just download. The existing per-PR benches in #### Upload / #### Download stay unchanged — those are PR-specific numbers documenting how each single change contributed; the new tables document the end-to-end 2.3.3-to-3.0.0 user experience. Numbers measured with the bench-matrix.sh dispatcher (scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2 manual aMule-vs-eMule runs. Raw logs in bench-results/bench-matrix-20260505T141651Z/.

Two tables at the top of the Performance section in 3.0.0: 1. Cross-platform end-to-end (2.3.3 vs 3.0.0) — sustained download throughput on macOS / Linux ARM / Windows ARM, peer-to-peer over LAN, 30 GB file: macOS: 0.35 -> 135 MB/s (381x) Linux: 0.34 -> 117 MB/s (345x) Windows: 0.36 -> 39 MB/s (107x) Plus a per-platform seeder/leecher contribution split that shows the seeder-side fix (#451) dominates and the leecher-side stack (#454/#484/#491) adds another 4.5-5.8x on top. 2. vs eMule 0.70b on Windows — same UTM hardware, both directions: Upload (Windows seeds -> Mac leecher): 22 vs 106 MB/s (~4.8x) Download (Linux seeds -> Windows leech): 20 vs 39 MB/s (~1.9x) Methodology footnote on the table since eMule has no remote- control protocol and is measured leecher-side via aMule's EC channel on the receiving box. Highlights line 16 also updated to mention both directions of the eMule comparison instead of just download. The existing per-PR benches in #### Upload / #### Download stay unchanged — those are PR-specific numbers documenting how each single change contributed; the new tables document the end-to-end 2.3.3-to-3.0.0 user experience. Numbers measured with the bench-matrix.sh dispatcher (scripts/bench-matrix.sh) — 12-run aMule-vs-aMule matrix plus 2 manual aMule-vs-eMule runs. Raw logs in bench-results/bench-matrix-20260505T141651Z/.

amuled's --full-daemon (-f) mode forks from inside InitGui() at amule.cpp:672. On POSIX, fork() only carries the calling thread to the child — any thread spawned before the fork has its OS-level pthread torn off, leaving the C++ object in memory with no thread actually running behind it. CPartFileWriteThread and CPartFileHashThread (both wxThread-backed, both spawned in their ctor) were constructed at amule.cpp:652-653, which is BEFORE the InitGui() fork. The comment at amule.cpp:682 explicitly flags this constraint for the same-shaped UploadBandwidthThrottler / CUploadDiskIOThread / CAsioService trio right next door — but the partfile-* threads, added later (#454), skipped it. User-visible symptom (#849): amuled -f reaches 99% on the network side, but the .part file stays at 0 bytes and CPU is pegged. The write-thread's PB_PENDING queue keeps filling because nothing is draining it on the child. Monolithic amule and amuled without -f both work because neither forks. The synchronous FlushBuffer fallback at PartFile.cpp:3252 doesn't save us either, because wxThread::IsRunning() reads an internal state flag that still says "running" post-fork — items get marked PB_PENDING and routed at the dead thread instead of being written inline. Fix: move the two constructions past InitGui(), into the same post-fork group as uploadDiskIOThread. No other code between the old and new construction sites touches either pointer (verified by grep in src/amule.cpp).

amule-project#454) On 32-bit targets the throttler's std::atomic<int64_t> needs libatomic (the ops expand to __atomic_*_8 library calls). The availability check used find_library(atomic), which only searches standard filesystem paths. With GCC, libatomic ships inside the compiler's own runtime dir (e.g. .../lib/gcc14/), which isn't on that path, so find_library false-fails and configure aborts -- even though `-latomic` links fine because the GCC driver resolves its internal copy (reported on MacPorts, amule-project#453). Replace find_library with check_library_exists(atomic __atomic_load_8), which links a probe with `-latomic` through the compiler driver. It succeeds wherever the flag actually links -- GCC's internal libatomic or a system libatomic (Clang / distro packages, versioned or not) -- and only errors when the flag genuinely can't link. The 32-bit "require libatomic" decision and the FATAL_ERROR guidance are unchanged; only the availability probe changes. Closes amule-project#453.

got3nks added 2 commits April 19, 2026 20:03

got3nks force-pushed the download-write-thread-pr branch from f1db159 to a200b1d Compare April 19, 2026 20:11

got3nks mentioned this pull request Apr 20, 2026

macOS: fix Dock right-click Quit skipping shutdown cleanup #456

Merged

mrjimenez merged commit 08c50d6 into amule-project:master Apr 22, 2026
6 checks passed

got3nks mentioned this pull request Apr 29, 2026

Defer per-part hash verification to a dedicated worker thread #498

Merged

This was referenced Apr 29, 2026

Downloades are faster when restarting aMule #372

Closed

release: propose 3.0.0-beta changelog (RFC) #521

Merged

got3nks deleted the download-write-thread-pr branch May 3, 2026 15:19

got3nks mentioned this pull request May 5, 2026

CHANGELOG: 3.0.0 throughput benchmark tables #528

Merged

This was referenced Jun 4, 2026

fix(amuled): construct partFile write/hash threads after InitGui fork (#849) #850

Merged

amuled not allocating downloads on raspberry pi, files never written to disk #849

Closed

Navigation Menu

Uh oh!

download: port eMule's CPartFileWriteThread to offload disk writes - #454

download: port eMule's CPartFileWriteThread to offload disk writes#454
mrjimenez merged 2 commits into
amule-project:masterfrom
got3nks:download-write-thread-pr

got3nks commented Apr 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

got3nks commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in this PR

1. Write thread

2. m_iWrites thread safety

3. Remote GUI build

Benchmarks

Client on master + server with #436, #451, #452, #453

Client with this PR (write thread) + server with #436, #451, #452, #453

Backward compatibility

Trade-offs

Files changed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

got3nks commented Apr 19, 2026 •

edited

Loading

2. `m_iWrites` thread safety