fix: purge stale drawers before re-mine to avoid hnswlib update-path race#523
fix: purge stale drawers before re-mine to avoid hnswlib update-path race#523StefanKremen wants to merge 1 commit intoMemPalace:developfrom
Conversation
…race Modified-file re-mines previously upserted over existing deterministic drawer_ids, which pushes ChromaDB through hnswlib.addItems -> addPoint(existing_internal_id) -> updatePoint -> repairConnectionsForUpdate. That path has a long-standing thread-safety bug in nmslib/hnswlib where searchBaseLayer dereferences a neighbor pointer being mutated by another worker thread. On macOS ARM64 with chromadb 0.6.3 + Python 3.13 this reliably segfaults the mining subprocess. This commit converts modified-file re-mines into a clean delete-by-source_file + insert, bypassing the update path entirely. Note: mcp_server.py::tool_add_drawer (line 364) does NOT need the same patch. Its drawer_id is content-keyed (sha256(wing+room+content[:100])) and there is a pre-check at lines 355-361 that short-circuits already-existing IDs before reaching upsert, so it only runs the pure INSERT path. Fixes MemPalace#521
|
Clean targeted fix for the hnswlib update race. The one-hunk approach (delete-before-upsert in The analysis on One test gap: the test plan marks the regression test as unchecked ( Also: #522 and #523 overlap on the same |
Delete existing drawers for a file before re-inserting fresh chunks. Converts re-mines from upsert (hnswlib updatePoint path, thread-unsafe on macOS ARM + chromadb 0.6.3) into delete+insert (safe addPoint path). Credit: @StefanKremen (#523) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
|
FYI: #251 already implements this purge-before-re-insert pattern in If #251 lands, this race is fixed as a side effect. If #523 lands first, the narrower scope here is fine and #251 just needs a rebase. |
|
Closing as superseded. The delete-before-upsert pattern proposed here landed upstream via a different PR — #784 (merged 2026-04-13, shipped v3.3.0) wrapped both miners' delete+insert critical sections in per-file Additional reinforcements since:
Closing without merge — the fix is in main, just not from this branch. |
Summary
Fixes the hnswlib
updatePoint/repairConnectionsForUpdaterace documented in #521. One-hunk patch inmempalace/miner.py::process_file: delete any drawers for asource_filebefore re-inserting the fresh chunks, so modified-file re-mines never enter the update path.Why only miner.py
mcp_server.py::tool_add_draweruses a content-keyed drawer_id and has an existence pre-check at lines 355-361 that short-circuits before the upsert. Combined with single-item batches per MCP tool call, that path is structurally incapable of reachingupdatePoint/repairConnectionsForUpdateunder normal operation. No parallel fix needed.Test plan
os.utime()the file, re-mine, assert no crash and new chunks replace oldpytest tests/ -v— full suite passes (534 passed, 106 deselected, 20.78s)ruff check mempalace/miner.py— cleanruff format --check mempalace/miner.py— cleantoucha file, re-mine — no segfault after patchNot included
mcp_server.py(analysis in the commit message).Fixes #521