Conversation
Same class of bug as #1007: ChromaDB's query() can return None in the documents and metadatas arrays when a drawer's HNSW vector entry exists but its metadata/document rows haven't been materialized. The code in Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...), meta.get("room", ...), meta.get("source_file", ...) directly without null safety, so it raises: AttributeError: 'NoneType' object has no attribute 'get' Two-line defensive coercion matching the pattern in #1009 / PR #999 for searcher.py: meta = meta or {}, doc = doc or "". The hit still appears with its real distance; source/wing/room fall back to their fallback values where the metadata row is missing. Frequently hit on chromadb 1.5.x (root cause #1006). Even after the chromadb floor lands (#1010), partial-state results remain possible during interrupted mines and schema upgrade boundaries, so the guard is worth having on its own. Fixes #1011.
ffe8e9a to
fa7fe1d
Compare
|
Rebased onto current develop (tip Both Legion345 commits preserved with original authorship through the rebase — only the SHAs changed ( |
igorls
left a comment
There was a problem hiding this comment.
Tested locally on a clean pr-1010 worktree. Open question in the PR body — whether chromadb 1.5.4 itself is clean on the #1006 queue-stall path — is now answered: yes.
What I ran
Python 3.14.4 venv, chromadb pinned to exactly ==1.5.4:
import chromadbsucceeds — no Pydantic V1ConfigError(issue #487).- End-to-end mine →
col.count()→ search, using a synthetic Claude-format JSONL session:Drawers filed: 2 collection count after mine: 2 ← matches, no #1006 queue stall search hits (first query): 4 ← hybrid expansion, all non-None meta - Full test suite: 991 passed, 2 warnings, 85s.
Python 3.14.4 venv, uv's resolved ==1.5.7 (same as lockfile):
- Same e2e: count matches, search works.
- Full test suite: 991 passed, 82s.
So >=1.5.4 is the right floor — it's where the queue-stall is already fixed, not just where Pydantic V1 was dropped. No need to tighten to 1.5.7+.
Logic-diff review (ignoring uv.lock)
Three-line production change is clean:
pyproject.toml:chromadb>=0.5.0→chromadb>=1.5.4,<2, plusPython 3.13/3.14classifiers. The<2cap is correct belt-and-suspenders per @web3guru888's feedback thread.mempalace/__init__.py: posthog telemetry-silencer comment updated to reflect that the filter is now a guard against regressions (chromadb 1.x makes posthog a no-op stub). Thelogging.getLogger(...).setLevel(CRITICAL)call is kept — good, harmless as a guard.mempalace/migrate.py: docstring updated to note chromadb 1.x auto-migrates 0.4.1+ palaces, and this command is now only for downgrades / auto-migrate failures. Accurate.
What I didn't test
- Existing-palace 0.6.x → 1.5.x auto-migration. The PR relies on @jphein's 50K-drawer validation in #581. Relying on upstream validation here is fine; I didn't spin up a 0.6.x palace to re-verify.
- Windows.
Attribution
Cherry-picked commits preserve @Legion345 as author per the PR body — confirmed via git log. Credit chain (Legion345 / web3guru888 / jphein) is spelled out clearly.
LGTM — happy to merge once CI is green. The open question about 1.5.4 is resolved; the pin is correct as written.
Same class of bug as #1007: ChromaDB's query() can return None in the documents and metadatas arrays when a drawer's HNSW vector entry exists but its metadata/document rows haven't been materialized. The code in Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...), meta.get("room", ...), meta.get("source_file", ...) directly without null safety, so it raises: AttributeError: 'NoneType' object has no attribute 'get' Two-line defensive coercion matching the pattern in #1009 / PR #999 for searcher.py: meta = meta or {}, doc = doc or "". The hit still appears with its real distance; source/wing/room fall back to their fallback values where the metadata row is missing. Frequently hit on chromadb 1.5.x (root cause #1006). Even after the chromadb floor lands (#1010), partial-state results remain possible during interrupted mines and schema upgrade boundaries, so the guard is worth having on its own. Fixes #1011.
…uard Merges MemPalace#990 (RFC 002 spec), MemPalace#1014 (BaseSourceAdapter/PalaceContext scaffolding), MemPalace#1013 (Layer3.search_raw None guard), MemPalace#1012 (docs), MemPalace#1010 (chromadb >=1.5.4), and MemPalace#998 (sweeper/tandem transcript safety net). Fork changes preserved: - quarantine_stale_hnsw() in chroma.py (guards HNSW/sqlite drift segfault) - get-then-create instead of get_or_create (guards ChromaDB 1.5.x metadata segfault) - paginated status() loop (guards SQLite variable limit on large palaces) - searcher hits-loop, BM25 fallback, _count_in_scope - .jsonl exempt from JUNK_FILE_SIZE cap (Claude Code transcripts can be large) - _validate_where() + operator constants taken from upstream Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Version bumps across pyproject.toml, mempalace/version.py, README badge, uv.lock, and plugin manifests (.claude-plugin/*, .codex-plugin/*). CHANGELOG aligned with main (post-3.3.1) and a new [3.3.2] section added covering the 11 PRs merged on develop since v3.3.1 — silent-transcript-drop fix + tandem sweeper (#998), None-metadata guards (#999, #1013), chromadb ≥1.5.4 for Py 3.13/3.14 (#1010), Windows Unicode (#681), HNSW quarantine recovery (#1000), PID stacking guard (#1023), doc-path cleanup (#996, #1012), and RFC 001/002 internal scaffolding (#995, #1014, #990).
Summary
Fixes #1006 (chromadb 1.5.x queue-stall regression) and #487 (chromadb 0.6.x fails on Python 3.13/3.14 due to Pydantic V1 incompatibility) via a single pin change:
```diff
```
Plus a regenerated `uv.lock` and two docstring/comment updates in `mempalace/init.py` and `mempalace/migrate.py` that reflect the new behavior (chromadb 1.x auto-migrates 0.4.1+ palaces; posthog telemetry is now a no-op stub).
Provenance
This is a maintainer-sponsored integration of @Legion345's work from #581. The two commits below carry Legion345 as the original git author, preserving their authorship unchanged. I'm landing this on a parallel branch to move the fix through the maintainer path while the #581 thread sits on community re-review after its Apr 17 rebase.
Credits:
Why this instead of the closed #1008
#1008 took the opposite direction — `chromadb<1.0`. A code review on that PR flagged that the `<1.0` cap reintroduces a documented breaking incompatibility with Python 3.13+ (see #487 for the exact Pydantic V1 `ConfigError`), reversing the intent of #302 which specifically widened chromadb to fix that.
Raising the floor to 1.5.4 is the correct direction: it excludes the broken 1.5.3-and-below versions (covering the known Pydantic V1 cutoff), keeps Python 3.13+ working, and adds a `<2` safety cap to guard future chromadb major breakage per @web3guru888's feedback thread.
Open question on the queue-stall floor
The #1006 reproduction tested chromadb 1.5.6 and 1.5.8 — both stall in the same way (embeddings queued, never materialized; search returns ghosts). I have NOT tested 1.5.4 against that reproduction.
If 1.5.4 is clean on the queue-stall path, this pin is sufficient as-is. If 1.5.4 also stalls, a follow-up tightens the floor (e.g., to the specific patch where the processor was fixed) on top of this PR — strictly narrower than the blanket `<1.0` approach, and no regression for Python 3.13+ users.
Relationship to #581
Happy to close this in favor of #581 if @Legion345 or the maintainers prefer — the diff is Legion345's verbatim. This PR exists as a sponsored merge path, not competing work.
Supersedes / closes
pip install mempalacesilently broken on chromadb 1.5.x — writes queued but never flush, search returns ghosts #1006 (queue-stall), chromadb<0.7 pin breaks MCP server on Python 3.14 #487 (Pydantic V1 on Py 3.13/3.14)Test plan
pip install mempalacesilently broken on chromadb 1.5.x — writes queued but never flush, search returns ghosts #1006 fix: `embeddings_queue` drains, `col.count()` matches filed count)