Review: Content Hash DB with Bloom Filter

This directly parallels what we built in our integration, so I can offer a detailed comparison.

Architecture Comparison

Your approach: SHA256 content hash → bloom filter fast-path → JSON-backed ContentHashDB → ChromaDB fallback
Our approach: Content hash (SHA256) → bloom filter pre-check → tiered cosine similarity dedup (hard=0.86, soft=0.55 thresholds) scoped to wing+room

The key difference: your approach catches exact content duplicates at the file level (same bytes = same hash), while ours catches semantic near-duplicates at the chunk level (different words, same meaning). Both are needed for production — yours is the I/O optimization layer, ours is the embedding quality layer.

Code Review

BloomFilter implementation — Clean. Using md5(item + str(i)) for hash functions is the standard "double hashing with string salt" approach. Reasonable defaults (100k capacity, 1% FPR).

check_and_add() correctness — Good that you verify against _hash_set (O(1) set lookup) rather than relying solely on the bloom filter. The bloom filter is correctly used as a fast negative check only. The fix note about this is appreciated.

⚠️ Concern — JSON serialization for the bloom filter:

json.dump({"array_size": self.size, "hash_count": self.hash_count, "array": self.array}, f)

With 100k capacity and 1% FPR, self.size = ~958,506 booleans. That's a 958k-element JSON array of true/false values → ~6-7 MB on disk. This works fine, but consider using a bitarray or struct.pack for the save/load path if performance becomes an issue at scale. For now it's fine — correctness over optimization.

⚠️ Concern — Race condition in mine() integration:
The check_and_add() call happens before processing, then flush() at the end. If the process crashes mid-mine, the in-memory bloom filter has entries that never persisted. On restart, those files will be re-processed (which is safe — worst case is redundant work). But if _save() happened to persist the JSON hashes dict without the bloom, the bloom becomes stale. Consider calling flush() periodically (e.g., every N files) rather than only at the end.

Missing: import json in miner.py — The diff adds import json to miner.py but I don't see it used there. The content_hash module has its own json import. Is the miner import leftover from an earlier iteration?

Test coverage — Good. test_different_files_same_content is the critical test for content-hash dedup and it's passing. The test_false_positive_handled test name is slightly misleading (it tests normal add flow, not actual false positive recovery), but the logic is sound.

Integration suggestion

This pairs well with #280's semantic dedup. Content hash catches byte-identical duplicates at I/O time (fast, O(1)), and semantic dedup catches near-identical content at embedding time (slower, but catches paraphrases). Consider noting this complementarity in the README.

🔭 Reviewed as part of the MemPalace-AGI integration project — autonomous research with perfect memory. Community interaction updates are posted regularly on the dashboard.