known2_64.met: orphan-prune AICH hashsets at sync startup - #605
Merged
mrjimenez merged 1 commit intoMay 14, 2026
Merged
Conversation
known2_64.met is the AICH (Advanced Intelligent Corruption Handling) hashset cache: a Merkle tree of SHA-1 hashes per shared file, used by peers to recover from sub-part-granularity corruption. Each entry is keyed by its AICH root hash; ngosang's amule-project#597 report had it at 1.1 GB. The file has no mtime-touch bloat (entries are content-addressed, dedup is already in place since amule-project#581), but it never shrinks: once a hashset is cached, the entry stays even after the underlying file leaves the user's library. After amule-project#598's known.met TTL prune drops ~14 k orphaned live entries on a long-lived profile, those hashes' AICH entries in known2_64.met are dead weight and should follow them out. Add CKnownFileList::CollectLiveAICHRoots() -- walks m_knownFileMap and m_duplicateFileList under list_mut, returns the set of AICH master hashes still referenced by either. Both lists need to be scanned: Append's demote branch parks a record (with its hashset) on the duplicate list while the new record takes the live slot, and an mtime-restore can re-promote the duplicate later. Dropping a duplicate's hashset would silently lose it on re-promote. Extend CAICHSyncTask::Entry()'s existing known2_64.met walk: open a "<name>.new" temp file via CFile::write_safe, and for each entry read from the source, either copy it through to the temp (if its root hash is in liveRoots) or skip it. On clean walk completion the Close() atomic-renames .new over the original; on corruption catch or IO error the temp is removed without finalising, leaving the source's existing truncation-recovery path intact. Hashset bytes are streamed through a 64 KB buffer rather than slurped, so a single large-file entry can't dominate the working set. The dedup root-hash cache (s_rootHashCache, amule-project#581) mirrored the old file; invalidate it after a non-zero drop so the next SaveHashSet rebuilds against the rewritten known2_64.met. Effective TTL is inherited from known.met: a record evicted there by PruneDuplicates ages out of liveRoots and its hashset gets dropped on the next AICH sync. Decoupled lifecycles would require bumping KNOWN2_MET_VERSION to add a per-entry timestamp (the file format is positional, not tag-based), which is out of scope here. Defensive: if knownfiles isn't yet populated (empty liveRoots), the prune is skipped -- we don't wipe everything on a misconfigured start.
got3nks
marked this pull request as ready for review
May 14, 2026 16:37
|
My known2.met is slightly over 3GB. It'll be interesting to see how much this shrinks |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #598 — closes the AICH side of @ngosang's #597 numbers.
known2_64.metis the AICH hashset cache (Merkle tree of SHA-1 hashes per shared file, used by peers for sub-part corruption recovery). It's content-addressed by AICH root hash, so it never had the mtime-touch bloat that hitknown.met. But it never shrinks either: once a hashset is cached, the entry stays even after the underlying file leaves the user's library. On ngosang's profile this was 1.1 GB, and after #598's TTL dropped ~14 k orphaned live entries fromknown.met, the corresponding AICH entries inknown2_64.metare now dead weight.The fix is a single-pass orphan prune in
CAICHSyncTask::Entry()(already walksknown2_64.metat startup to build the hashlist forCheckAICHHashes). Snapshot the set of AICH master hashes still referenced by a live or duplicate-listCKnownFile(CKnownFileList::CollectLiveAICHRoots, added), then while walking source entries, stream-copy each one into a<name>.newtemp file only if its root hash is in the live set. On clean walk completion,CFile::write_safe'sClose()atomic-renames.newover the original; on corruption or IO failure the temp is removed without finalising, preserving the existing truncation-recovery path.Hashset bytes are streamed through a 64 KB buffer rather than slurped — a single large-file entry can have megabytes of SHA-1s and we don't want one to dominate the working set.
Effective TTL is inherited from known.met. When
PruneDuplicatesevicts a known.met record for being unseen for 30 d, its AICH master hash leaves the live set, and the next AICH sync drops the orphaned hashset. Decoupled lifecycles would require bumpingKNOWN2_MET_VERSIONto add a per-entry timestamp (the file format is positional, no tag system), which is deliberately out of scope here.Defensive: if
theApp->knownfilesisn't populated yet (empty liveRoots set), the prune is skipped — we don't wipe everything on a misconfigured start.Both lists (
m_knownFileMapANDm_duplicateFileList) are scanned inCollectLiveAICHRoots:Append's demote branch parks a record (with its hashset) on the duplicate list while the new record takes the live slot, and an mtime-restore can re-promote the duplicate. Dropping a duplicate's hashset would silently lose it on re-promote.The dedup root-hash cache (
s_rootHashCache, #581) mirrored the old file; invalidated after a non-zero drop so the nextSaveHashSetrebuilds against the rewrittenknown2_64.met.Kept as draft while @ngosang validates the eviction behaviour against his 1.1 GB profile.