SharedFileList: drop O(N*M) AICH-hash scan to O(N+M) (refs #745) - #746
Merged
Merged
Conversation
The startup sync task hands CheckAICHHashes a std::list<CAICHHash> loaded from known2_64.met and the function does a linear std::find over that list for every shared file, all under list_mut. On sharesets of 100 000+ files (issue amule-project#745) the O(N*M) walk holds the mutex long enough that the watcher's debounced Reload() on the GUI thread is queued behind it, freezing the UI for minutes. Build an unordered_set<CAICHHash> from the input once and probe it per file. std::hash<CAICHHash> is already specialised in SHAHashSet.h, so each lookup is O(1) average and the total drops from O(N*M) to O(N+M). The list mutex is taken only after the index is built, so no extra critical-section time is added.
got3nks
force-pushed
the
pr-checkaich-hashset
branch
from
May 27, 2026 15:36
a289e4a to
a30e9b7
Compare
4 tasks
1 task
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Jun 4, 2026
…ndex Adds 55+ merged PRs to the 3.0.0 changelog since the last update (amule-project#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796 (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths). - Bug Fixes & Stability: post-amule-project#744 fixes including EC notification leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728), amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760), PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788, amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756), Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833, amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774). - Translations: late-cycle wave covering French/Turkish manpages (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for date+version drift (amule-project#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
got3nks
added a commit
to got3nks/amule
that referenced
this pull request
Jun 4, 2026
…ndex Adds 55+ merged PRs to the 3.0.0 changelog since the last update (amule-project#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796 (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths). - Bug Fixes & Stability: post-amule-project#744 fixes including EC notification leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728), amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760), PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788, amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756), Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833, amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774). - Translations: late-cycle wave covering French/Turkish manpages (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for date+version drift (amule-project#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
mrjimenez
pushed a commit
that referenced
this pull request
Jun 4, 2026
Adds 55+ merged PRs to the 3.0.0 changelog since the last update (#747, 2026-05-27). Narrative additions cover: - Packaging: expanded the top list to include the macOS per-arch .app bundles and the Windows NSIS installer alongside the existing AppImage / Flatpak / .dmg / .zip entries. New bullets for #785 (alc/alcc/cas/wxcas everywhere + Windows amuleweb), #794 (.dmg amuleweb path), #789 (<OS>-<arch> artifact naming), #780 / #796 (Windows DPI + comctl32 manifest), #784 (FHS share/amule paths). - Bug Fixes & Stability: post-#744 fixes including EC notification leak (#797), big-library scaling (#736, #840 superseding #728), amulegui ghost entries (#810, #819, #841, #824, #830, #760), PartFile early hash (#762), server protocol fixes (#835, #788, #721, #787), crypto stream UB (#779), UAF prevention (#756), Kad rotation (#795, #799/#805), GTK warning silencing (#833, #826/#836), and the clang-tidy worklist (#770, #772-#774). - Translations: late-cycle wave covering French/Turkish manpages (#753/#754/#776), Galician (#763), Slovenian (#771), pt-BR (#768/#775/#812), French (#811), plus man-page tooling for date+version drift (#802). - Contributors: added ngosang for UX feedback on the late-3.0 cycle (#817/#818/#821/#828/#844) and ongoing work on the user-facing manual at amule-org.github.io. - Merged PRs flat index: extended with #746-#845 + #841.
mrjimenez
pushed a commit
to mrjimenez/amule
that referenced
this pull request
Aug 1, 2026
amule-project#746) * build: require crypto++ 8.1 and drop the hand-rolled HKDF The 5.6 floor dated from before the EC encryption work and had started to cost more than it bought. It forced ChaCha20-Poly1305 behind a version gate, and it forced HKDF to be written by hand, because cryptopp's own hkdf.h only appeared in 5.6.3. Hand-rolled crypto is worth avoiding when the library already has it, so with the floor at 8.1 both go away: ChaCha is compiled in unconditionally, and HkdfSha256 becomes a thin wrapper over CryptoPP::HKDF. The RFC 5869 vector in ECCryptTest passes unchanged, which is what proves the two implementations agree byte for byte -- had they differed, every derived key would have changed silently. Cipher negotiation stays: that is about what the peer can do, not what this build has. 8.1 is a conservative floor. Nothing in circulation is below it -- Debian oldstable carries 8.4, and everything more recent is 8.6 or newer. The cryptopp-modern fork is unaffected, since it is based on 8.9.0 and the cmake check exempts its calendar versioning. * fix(security): restrict the config files to owner-only amule.conf and remote.conf hold the EC password, and in this protocol the stored value *is* the credential -- anything that can read the file can drive the daemon, without inverting any hash. They are created with whatever the umask allows: 0644 on macOS, and 0664 under the 0002 umask Debian and Ubuntu ship, which leaves the file group-writable. Tighten to 0600 at startup rather than at creation, so configs that already exist are fixed on the first run of this version and not just new installs. Verified that wxFileConfig replaces the file on save but carries the mode across, on both macOS and Linux, so one call at startup holds for the life of the install. amule.conf.bak is a full copy including the password and gets the same treatment. The helper returns whether it changed anything instead of logging: mulecommon sits below the logger, and the caller can say it once. No-op on Windows, which has no POSIX mode bits -- there the file is protected by the profile directory's ACL, the same compromise Credentials.cpp already makes. * chore(i18n): regenerate catalogs for the permissions notice * fix(security): also restrict the config on a fresh install The startup pass runs before wxFileConfig has created anything, so on a clean install there was no file to tighten and the config sat at the umask default -- 0644 on macOS, 0664 and group-writable on Debian and Ubuntu -- for the whole first session, until the next start. Tighten again right after the first Flush(), once the file exists. Cheap to repeat: RestrictToOwner stats first and does nothing when the mode is already owner-only, which it stays, because wxFileConfig carries the mode across the replace it performs on every later save. * fix(security): cover remote.conf on a fresh amulegui install The post-Flush pass hardcoded "amule.conf", but Preferences.cpp is compiled into amulegui too, where the config is remote.conf. So a fresh amulegui install chmod'ed a file that does not exist and left its actual config at the umask default -- 0664 and group-writable on Debian and Ubuntu -- until the next start. The startup pass already used m_configFile and was correct; the two agreed only by coincidence on the core. Both now derive from theApp->m_configFile, so there is one source of truth and the .bak follows it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The startup AICH sync task (
CAICHSyncTask::Entry) handsCSharedFileList::CheckAICHHashesastd::list<CAICHHash>loaded fromknown2_64.met, and for every shared file the function does a linearstd::findover that list — all while holdinglist_mut. On sharesets of 100 000+ files (issue #745) the O(N×M) scan keeps the mutex long enough that the watcher's debouncedReload()on the GUI thread blocks behind it for minutes.The fix builds an
unordered_set<CAICHHash>from the input once, then probes it per file.std::hash<CAICHHash>is already specialised inSHAHashSet.h, so each lookup is O(1) average and the total drops from O(N×M) to O(N+M). The set is constructed before the lock is taken, so no extra critical-section time is added.Single-file change. No API change (
CheckAICHHasheskeeps the same signature).Reference
gdb backtrace from #745 — Thread 1 (GUI) blocked on
wxMutex::LockinsideCSharedFileList::FindSharedFileswhile Thread 2 (CTaskThread) actively runs inside the linearstd::findloop ofCheckAICHHashesholdinglist_mut. Pinpoints the contention exactly to this loop.Test plan
amuledandamuletargets, wxGTK 3.2.9 / Boost 1.90)amuled --versionruns (rev. 2.3.3-504-ga289e4abc)known2_64.met,Cat_AICH-Hasherverbose logging): both binaries report identical log output — "Trovati 50 file condivisi conosciuti", "AICH-Hasher: Synchronization thread started", "Masterhashes of known files have been loaded", zero "Hashing requested" lines (every known hash matched the index in both builds)