fix: upgrade chromadb to >=1.5.4 for python 3.13/3.14 compatibility#581
fix: upgrade chromadb to >=1.5.4 for python 3.13/3.14 compatibility#581Legion345 wants to merge 0 commit intoMemPalace:developfrom
Conversation
web3guru888
left a comment
There was a problem hiding this comment.
The fix is correct for Python 3.13/3.14, and the diagnosis is right — Pydantic V1 is the blocker, and 1.5.4 is where chromadb fully dropped it.
A few things worth flagging before merge:
Version skip: Going from <0.7 to >=1.5.4 skips the entire 0.7–1.5.3 range. That's intentional (0.7–1.5.3 still has partial Pydantic V1 usage or other issues), but the pin is now unbounded upward — a chromadb 2.0 with breaking API changes would be picked up automatically. Worth considering >=1.5.4,<2 as a safety valve unless there's confidence the API won't break.
Existing 3.1.0 users: Anyone who installed mempalace 3.1.0 has a chromadb 0.6.x palace. The mempalace migrate command handles this, and chromadb 1.x does auto-migrate on first open. But upgrading from 3.1.0 → 3.2.0 will silently modify the user's palace directory on first use. That's probably fine, but a release note / upgrade guide entry would help prevent surprise.
The <0.7 pin context: According to the commit history, the pin was in place at 3.1.0. Dependabot later opened a PR to bump to <1.6 — do you know why that wasn't merged? If there were migration issues discovered then, worth knowing whether 1.5.4 is actually safe.
The code change itself is minimal and correct (3 non-lock files, all documentation/comment updates rather than logic changes). If the CI is green and the auto-migration path is validated, this seems mergeable.
[MemPalace-AGI integration — 215 tests, 710 KG entities]
web3guru888
left a comment
There was a problem hiding this comment.
The Python 3.13/3.14 Pydantic V1 incompatibility is a real blocker and the version boundary (>=1.5.4) is the right cut. Pydantic V1 compatibility in chromadb was fully dropped in the 1.x series and the 1.5.x release specifically cleaned up the remaining shims.
A few notes on the version pin:
Upper bound: >=1.5.4 with no upper bound means future chromadb major versions (2.x, 3.x) will be picked up automatically. Given chromadb's history of API-breaking changes between minor versions, a bounded range like >=1.5.4,<2.0 would be safer until the maintainers validate against 2.x. The current PR removes the bound entirely, which is the opposite failure mode from the current <0.7 over-constraint.
Migration handling: If existing users have palaces created with chromadb 0.6.x, the upgrade path matters. chromadb's internal format has changed between 0.x and 1.x — the PR mentions "chromadb handles migrating existing databases automatically" but this is worth testing explicitly, particularly for users with large palaces. The migrate.py file in the changeset presumably handles this — worth confirming it's documented in the PR.
Pydantic V2 compat in mempalace code: If mempalace itself uses any pydantic models directly (not just through chromadb), those would also need to be V2-compatible. Likely not an issue here but worth confirming.
The core fix is right. The upper bound concern is the main thing to address before merge.
[MemPalace-AGI integration — 215 tests, 710 KG entities]
web3guru888
left a comment
There was a problem hiding this comment.
The Pydantic V1 compat problem is real and this fix is directionally correct — chromadb>=1.5.4 resolves the Python 3.13/3.14 incompatibility and is the right target version.
A few things worth flagging before merge:
Existing migration story: The 3.1.0 pin to chromadb>=0.5.0,<0.7 was intentional — users on 3.0.0 (which used 1.5.x) had their on-disk format broken when 3.1.0 downgraded to 0.6.x. This PR reverses that by jumping back to 1.5.x. The updated migrate.py docstring correctly notes that chromadb handles 0.4.1+ automatic migration on first open, so forward migration should be clean. But users who ran 3.1.0 with 0.6.x databases and are now upgrading to 3.2.0 with 1.5.4+ will go through a database migration on first open — worth calling out explicitly in a CHANGELOG entry or release note.
Upper bound: chromadb>=1.5.4 with no upper bound means future chromadb 2.x (if that happens) could break things silently. >=1.5.4,<2 would be safer for now, or at least document the intentional open-ended range.
Test coverage on 3.13/3.14: The classifiers are added, but is CI actually running the test matrix against Python 3.13 or 3.14? Worth verifying the .github/workflows/ci.yml includes those versions.
The uv.lock diff is expected given the dependency change. The 946 additions in the lockfile are a chromadb 1.5.x transitive dependency tree.
|
This is the right fix long-term. @robottwo confirmed in #521 that chromadb 1.5.7 resolves both the ARM64 HNSW destructor segfault and the search crash entirely — no workarounds needed. Our fork (#562) currently works around the HNSW corruption with a nuke-and-rebuild repair command and inode-based MCP reconnection, but those are recovery tools, not a root cause fix. We'll test the 1.5.4+ upgrade against our 50K-drawer palace and likely adopt it. One thing to verify: does chromadb 1.5.4+ auto-migrate existing |
Fixes ARM64 HNSW segfaults, Python 3.13/3.14 compat. Auto-migrates existing databases, zero API breakage. 648 tests pass, 50K palace verified. Co-Authored-By: Claude Opus 4.6 <[email protected]>
|
Tested against our 50K-drawer palace — chromadb 1.5.7 auto-migrated the 0.6.3 database seamlessly. Zero API breakage, zero test failures (648 pass), query results identical. Also 20% faster test suite execution (31s vs 38s). Adopted in #562 — bumped To answer my own question: existing databases auto-migrate on first read with no manual step needed. Clean upgrade. |
|
@jphein — that's the confirmation I was waiting for. Seamless 0.6.3 → 1.5.7 auto-migration with zero API breakage and 648 tests passing is exactly what removes the upper-bound concern I raised in my initial review. Changing my position: this PR should merge as-is once milla-jovovich is ready. The migration story resolves itself. The 20% test suite speedup is a nice bonus. (My earlier comment flagging the upper-bound concern still stands as context, but it's no longer a blocker given jphein's validation at scale.) |
web3guru888
left a comment
There was a problem hiding this comment.
Updating to APPROVED based on jphein's production validation: 0.6.3→1.5.7 auto-migration seamless, 648 tests pass, 20% speedup. Upper-bound concern resolved.
|
Added the <2 cap. Good call. @jphein's migration test gave me confidence the upgrade path is clean. |
…air fix, chromadb upgrade - Add SKIP_PATTERNS and JUNK_FILE_SIZE to filter noise during mining (MemPalace#587) - Fix dry-run room=None crash with fallback to 'general' (MemPalace#586) - Fix status 10K ceiling with paginated offset loop (MemPalace#478) - Add mempalace purge command for targeted drawer removal - Fix repair to nuke palace dir for clean HNSW index rebuild - Upgrade chromadb pin to >=1.5.4 (MemPalace#581) Co-Authored-By: Claude Opus 4.6 <[email protected]>
ORT_DISABLE_COREML is not a recognized ONNX Runtime environment variable. ONNX Runtime does not expose a global env var to disable individual execution providers -- providers are selected per session via the providers argument to InferenceSession. Setting it had zero effect. The mitigation was added in df33550 (v3.1.0) with the stated goal of fixing the MemPalace#74 ARM64 segfault. Two problems: the env var doesn't work as described above, and MemPalace#74 is a null-pointer crash in chromadb_rust_bindings.abi3.so -- not an ONNX issue, so disabling CoreML would not have fixed it anyway. MemPalace#521 has since traced the actual macOS arm64 crashes (both mine and search) to the 0.x chromadb hnswlib binding. Filtering CoreMLExecutionProvider at the ONNX layer leaves the hnswlib C++ crash intact, so the real fix is upgrading chromadb to 1.5.4+, which MemPalace#581 proposes. This PR only removes the misleading no-op and leaves a NOTE pointing at MemPalace#521 / MemPalace#581. Closes MemPalace#397
ORT_DISABLE_COREML is not a recognized ONNX Runtime environment variable. ONNX Runtime does not expose a global env var to disable individual execution providers -- providers are selected per session via the providers argument to InferenceSession. Setting it had zero effect. The mitigation was added in df33550 (v3.1.0) with the stated goal of fixing the #74 ARM64 segfault. Two problems: the env var doesn't work as described above, and #74 is a null-pointer crash in chromadb_rust_bindings.abi3.so -- not an ONNX issue, so disabling CoreML would not have fixed it anyway. #521 has since traced the actual macOS arm64 crashes (both mine and search) to the 0.x chromadb hnswlib binding. Filtering CoreMLExecutionProvider at the ONNX layer leaves the hnswlib C++ crash intact, so the real fix is upgrading chromadb to 1.5.4+, which #581 proposes. This PR only removes the misleading no-op and leaves a NOTE pointing at #521 / #581. Closes #397
9a6f3a8 to
a145b0e
Compare
Legion345
left a comment
There was a problem hiding this comment.
Rebased onto upstream/main (v3.3.0) to resolve conflicts in pyproject.toml and uv.lock. No change to the actual diff — same two commits, new base. Re-approval appreciated when you get a chance.
a145b0e to
109d7f2
Compare
What does this PR do?
Fixes mempalace being completely broken on Python 3.13/3.14. The chromadb<0.7 pin pulls in chromadb 0.6.x, which uses Pydantic V1 — and Pydantic V1 doesn't work on Python 3.14. You get this on import:
pydantic.v1.errors.ConfigError: unable to infer type for attribute "chroma_server_nofile"
Bumping to >=1.5.4 fixes it. That's the specific version where chromadb dropped Pydantic V1 for good — anything between 1.0 and 1.5.3 still has the same problem. The rest of the API is unchanged so nothing else needed updating, and chromadb handles migrating existing databases automatically.
How to test
On Python 3.13 or 3.14:
pip install .
python3 -c "import mempalace; print('OK')"
Checklist