Skip to content

fix: upgrade chromadb to >=1.5.4 for Python 3.13/3.14 compatibility + fix 1.5.x queue-stall (closes #1006)#1010

Merged
igorls merged 2 commits intodevelopfrom
fix/chromadb-1-5-4-py-3-13-compat-via-581
Apr 18, 2026
Merged

fix: upgrade chromadb to >=1.5.4 for Python 3.13/3.14 compatibility + fix 1.5.x queue-stall (closes #1006)#1010
igorls merged 2 commits intodevelopfrom
fix/chromadb-1-5-4-py-3-13-compat-via-581

Conversation

@bensig
Copy link
Copy Markdown
Collaborator

@bensig bensig commented Apr 18, 2026

Summary

Fixes #1006 (chromadb 1.5.x queue-stall regression) and #487 (chromadb 0.6.x fails on Python 3.13/3.14 due to Pydantic V1 incompatibility) via a single pin change:

```diff

  • "chromadb>=0.5.0",
  • "chromadb>=1.5.4,<2",
    ```

Plus a regenerated `uv.lock` and two docstring/comment updates in `mempalace/init.py` and `mempalace/migrate.py` that reflect the new behavior (chromadb 1.x auto-migrates 0.4.1+ palaces; posthog telemetry is now a no-op stub).

Provenance

This is a maintainer-sponsored integration of @Legion345's work from #581. The two commits below carry Legion345 as the original git author, preserving their authorship unchanged. I'm landing this on a parallel branch to move the fix through the maintainer path while the #581 thread sits on community re-review after its Apr 17 rebase.

Credits:

  • @Legion345 — authored both commits; diagnosed the Pydantic V1 / Python 3.13/3.14 incompatibility and chose 1.5.4 as the correct floor (commits preserved via cherry-pick)
  • @web3guru888 — approved in #581 on Apr 11 after substantive review: "The fix is correct for Python 3.13/3.14, and the diagnosis is right — Pydantic V1 is the blocker, and 1.5.4 is where chromadb fully dropped it."
  • @jphein — production-validated the 0.6.3 → 1.5.7 auto-migration on a 50K-drawer palace: 648 tests pass, 20% test-suite speedup, zero API breakage

Why this instead of the closed #1008

#1008 took the opposite direction — `chromadb<1.0`. A code review on that PR flagged that the `<1.0` cap reintroduces a documented breaking incompatibility with Python 3.13+ (see #487 for the exact Pydantic V1 `ConfigError`), reversing the intent of #302 which specifically widened chromadb to fix that.

Raising the floor to 1.5.4 is the correct direction: it excludes the broken 1.5.3-and-below versions (covering the known Pydantic V1 cutoff), keeps Python 3.13+ working, and adds a `<2` safety cap to guard future chromadb major breakage per @web3guru888's feedback thread.

Open question on the queue-stall floor

The #1006 reproduction tested chromadb 1.5.6 and 1.5.8 — both stall in the same way (embeddings queued, never materialized; search returns ghosts). I have NOT tested 1.5.4 against that reproduction.

If 1.5.4 is clean on the queue-stall path, this pin is sufficient as-is. If 1.5.4 also stalls, a follow-up tightens the floor (e.g., to the specific patch where the processor was fixed) on top of this PR — strictly narrower than the blanket `<1.0` approach, and no regression for Python 3.13+ users.

Relationship to #581

Happy to close this in favor of #581 if @Legion345 or the maintainers prefer — the diff is Legion345's verbatim. This PR exists as a sponsored merge path, not competing work.

Supersedes / closes

Test plan

bensig added a commit that referenced this pull request Apr 18, 2026
Same class of bug as #1007: ChromaDB's query() can return None in the
documents and metadatas arrays when a drawer's HNSW vector entry exists
but its metadata/document rows haven't been materialized. The code in
Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...),
meta.get("room", ...), meta.get("source_file", ...) directly without
null safety, so it raises:

  AttributeError: 'NoneType' object has no attribute 'get'

Two-line defensive coercion matching the pattern in #1009 /
PR #999 for searcher.py: meta = meta or {}, doc = doc or "".
The hit still appears with its real distance; source/wing/room
fall back to their fallback values where the metadata row is missing.

Frequently hit on chromadb 1.5.x (root cause #1006). Even after the
chromadb floor lands (#1010), partial-state results remain possible
during interrupted mines and schema upgrade boundaries, so the guard
is worth having on its own.

Fixes #1011.
@bensig bensig force-pushed the fix/chromadb-1-5-4-py-3-13-compat-via-581 branch from ffe8e9a to fa7fe1d Compare April 18, 2026 19:06
@bensig
Copy link
Copy Markdown
Collaborator Author

bensig commented Apr 18, 2026

Rebased onto current develop (tip 2b9f17c) to pick up the RFC 001 backend entry-points block ([project.entry-points."mempalace.backends"]) from PR #995 which merged after this branch was originally based. Branch was ~4 commits behind.

Both Legion345 commits preserved with original authorship through the rebase — only the SHAs changed (8445b95d0c8ecd, ffe8e9afa7fe1d). No diff conflicts.

Copy link
Copy Markdown
Member

@igorls igorls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally on a clean pr-1010 worktree. Open question in the PR body — whether chromadb 1.5.4 itself is clean on the #1006 queue-stall path — is now answered: yes.

What I ran

Python 3.14.4 venv, chromadb pinned to exactly ==1.5.4:

  • import chromadb succeeds — no Pydantic V1 ConfigError (issue #487).
  • End-to-end mine → col.count() → search, using a synthetic Claude-format JSONL session:
    Drawers filed:                2
    collection count after mine:  2   ← matches, no #1006 queue stall
    search hits (first query):    4   ← hybrid expansion, all non-None meta
    
  • Full test suite: 991 passed, 2 warnings, 85s.

Python 3.14.4 venv, uv's resolved ==1.5.7 (same as lockfile):

  • Same e2e: count matches, search works.
  • Full test suite: 991 passed, 82s.

So >=1.5.4 is the right floor — it's where the queue-stall is already fixed, not just where Pydantic V1 was dropped. No need to tighten to 1.5.7+.

Logic-diff review (ignoring uv.lock)

Three-line production change is clean:

  • pyproject.toml: chromadb>=0.5.0chromadb>=1.5.4,<2, plus Python 3.13/3.14 classifiers. The <2 cap is correct belt-and-suspenders per @web3guru888's feedback thread.
  • mempalace/__init__.py: posthog telemetry-silencer comment updated to reflect that the filter is now a guard against regressions (chromadb 1.x makes posthog a no-op stub). The logging.getLogger(...).setLevel(CRITICAL) call is kept — good, harmless as a guard.
  • mempalace/migrate.py: docstring updated to note chromadb 1.x auto-migrates 0.4.1+ palaces, and this command is now only for downgrades / auto-migrate failures. Accurate.

What I didn't test

  • Existing-palace 0.6.x → 1.5.x auto-migration. The PR relies on @jphein's 50K-drawer validation in #581. Relying on upstream validation here is fine; I didn't spin up a 0.6.x palace to re-verify.
  • Windows.

Attribution

Cherry-picked commits preserve @Legion345 as author per the PR body — confirmed via git log. Credit chain (Legion345 / web3guru888 / jphein) is spelled out clearly.

LGTM — happy to merge once CI is green. The open question about 1.5.4 is resolved; the pin is correct as written.

bensig added a commit that referenced this pull request Apr 18, 2026
Same class of bug as #1007: ChromaDB's query() can return None in the
documents and metadatas arrays when a drawer's HNSW vector entry exists
but its metadata/document rows haven't been materialized. The code in
Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...),
meta.get("room", ...), meta.get("source_file", ...) directly without
null safety, so it raises:

  AttributeError: 'NoneType' object has no attribute 'get'

Two-line defensive coercion matching the pattern in #1009 /
PR #999 for searcher.py: meta = meta or {}, doc = doc or "".
The hit still appears with its real distance; source/wing/room
fall back to their fallback values where the metadata row is missing.

Frequently hit on chromadb 1.5.x (root cause #1006). Even after the
chromadb floor lands (#1010), partial-state results remain possible
during interrupted mines and schema upgrade boundaries, so the guard
is worth having on its own.

Fixes #1011.
@igorls igorls merged commit 7af3bfa into develop Apr 18, 2026
7 checks passed
jphein added a commit to jphein/mempalace that referenced this pull request Apr 19, 2026
…uard

Merges MemPalace#990 (RFC 002 spec), MemPalace#1014 (BaseSourceAdapter/PalaceContext scaffolding),
MemPalace#1013 (Layer3.search_raw None guard), MemPalace#1012 (docs), MemPalace#1010 (chromadb >=1.5.4),
and MemPalace#998 (sweeper/tandem transcript safety net).

Fork changes preserved:
- quarantine_stale_hnsw() in chroma.py (guards HNSW/sqlite drift segfault)
- get-then-create instead of get_or_create (guards ChromaDB 1.5.x metadata segfault)
- paginated status() loop (guards SQLite variable limit on large palaces)
- searcher hits-loop, BM25 fallback, _count_in_scope
- .jsonl exempt from JUNK_FILE_SIZE cap (Claude Code transcripts can be large)
- _validate_where() + operator constants taken from upstream

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
igorls added a commit that referenced this pull request Apr 19, 2026
Version bumps across pyproject.toml, mempalace/version.py, README badge,
uv.lock, and plugin manifests (.claude-plugin/*, .codex-plugin/*).

CHANGELOG aligned with main (post-3.3.1) and a new [3.3.2] section added
covering the 11 PRs merged on develop since v3.3.1 — silent-transcript-drop
fix + tandem sweeper (#998), None-metadata guards (#999, #1013),
chromadb ≥1.5.4 for Py 3.13/3.14 (#1010), Windows Unicode (#681),
HNSW quarantine recovery (#1000), PID stacking guard (#1023), doc-path
cleanup (#996, #1012), and RFC 001/002 internal scaffolding (#995, #1014, #990).
@igorls igorls mentioned this pull request Apr 19, 2026
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: fresh pip install mempalace silently broken on chromadb 1.5.x — writes queued but never flush, search returns ghosts

3 participants