Skip to content

feat: add Qdrant as alternative storage backend#700

Open
RobertoGEMartin wants to merge 1 commit intoMemPalace:developfrom
RobertoGEMartin:feat/qdrant-backend
Open

feat: add Qdrant as alternative storage backend#700
RobertoGEMartin wants to merge 1 commit intoMemPalace:developfrom
RobertoGEMartin:feat/qdrant-backend

Conversation

@RobertoGEMartin
Copy link
Copy Markdown

Summary

Adds Qdrant as an opt-in alternative to ChromaDB for the palace vector store. This resolves scalability issues when palaces exceed ~100K drawers (related to #688).

  • New QdrantBackend + QdrantCollection in mempalace/backends/qdrant.py, implementing BaseCollection
  • Backend selection via config.json: {"backend": "qdrant"} (ChromaDB remains the default)
  • Full ChromaDB filter compatibility: $eq, $ne, $in, $nin, $and, $or, $contains
  • Deterministic UUID5 ID mapping for reproducible migrations
  • Lazy imports — qdrant-client and sentence-transformers only loaded when Qdrant backend is active

Why Qdrant?

Metric ChromaDB Qdrant
Max tested drawers ~100K (SQLite variable limit) 168K+ (production-tested)
Storage (168K drawers) ~3.4 GB ~800 MB (~4x smaller)
Filter queries at scale Degrades (col.get() fails) Stable (payload indexes)

Installation

pip install mempalace[qdrant]

Then in ~/.mempalace/config.json:

{"backend": "qdrant"}

Changes

File Change
mempalace/backends/qdrant.py New: QdrantBackend + QdrantCollection (411 lines)
mempalace/backends/__init__.py Conditional Qdrant exports
mempalace/palace.py Config-driven backend selection
mempalace/config.py Added backend property
pyproject.toml Added qdrant optional dependency
tests/test_backends_qdrant.py 20 tests (unit + integration)
README.md Qdrant backend documentation

Test plan

  • All 20 Qdrant tests pass (17 unit + 3 integration)
  • All 6 existing ChromaDB backend tests still pass (zero regressions)
  • Ruff lint passes on all changed files
  • Production-tested with 168K+ drawers across 8 wings
  • CI pipeline validation

🤖 Generated with Claude Code

Adds QdrantBackend as an opt-in alternative to ChromaDB, selectable via
config.json. Scales beyond 100K drawers with ~4x less storage. Resolves MemPalace#688.

Install with: pip install mempalace[qdrant]
Enable with: {"backend": "qdrant"} in config.json

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Copy link
Copy Markdown
Member

@igorls igorls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, let's hold it until v4 lands

igorls added a commit that referenced this pull request Apr 12, 2026
Formalizes the BaseCollection/BaseBackend contract introduced as a seam
in #413 into an interchangeability spec that third-party backends can
build to. Driven by six in-flight backend PRs (#574, #643, #665, #697,
#700, #381) each implementing the interface differently.

Key decisions captured: entry-point distribution, typed QueryResult/
GetResult replacing Chroma dict shape, daemon-first multi-palace model
via PalaceRef, required where-clause subset (incl. $contains),
mandatory embedder injection with model-identity validation, capability
tokens, shared pytest conformance suite, and a backend-neutral
migrate/verify CLI.
igorls added a commit that referenced this pull request Apr 18, 2026
…nd registry (RFC 001 §10)

Advances RFC 001 §10 cleanup so backend-author PRs (#574 LanceDB, #665 Postgres,
#700 Qdrant, #697 hosted, #643 PalaceStore, #381 Qdrant) have a stable target
to align against.

Scope (this PR):

- Typed QueryResult / GetResult dataclasses replace Chroma's dict shape at
  the BaseCollection boundary (§1.3). A transitional _DictCompatMixin keeps
  existing callers working while the attribute-access migration proceeds.
- BaseCollection is now kwargs-only across add/upsert/query/get/delete/update
  with ABC defaults for estimated_count/close/health and a non-atomic default
  update() (§1.1–1.2).
- PalaceRef replaces raw path strings at the backend boundary (§2.2).
- BaseBackend ABC with get_collection/close_palace/close/health/detect (§2.3).
- mempalace.backends entry-point group + in-tree registry with
  resolve_backend_for_palace priority order matching §3.2–3.3.
- ChromaCollection normalizes chroma returns into typed results; unknown
  where-clause operators raise UnsupportedFilterError (no silent drop, §1.4).
- ChromaBackend absorbs the inode/mtime client-cache freshness check
  previously duplicated in mcp_server._get_client() (§10 + PR #757).
- searcher.py migrated to typed-attribute access as the reference call
  site; remaining callers land in a follow-up.
- pyproject: chroma registered via [project.entry-points."mempalace.backends"].

Out of scope (explicit follow-ups):

- Full caller migration off the dict-compat shim across palace.py,
  mcp_server.py, miner.py, convo_miner.py, dedup.py, repair.py, exporter.py,
  palace_graph.py, cli.py, closet_llm.py.
- Embedder injection + three-state EmbedderIdentityMismatchError check (§1.5).
- maintenance_state() / run_maintenance() benchmark hooks (§7.3).
- AbstractBackendContractSuite full coverage (§7.1–7.2).
- mempalace migrate / mempalace verify CLI rewrites through BaseCollection (§8).

Tests: 970 passed (up from 967 on develop); new coverage for typed results,
empty-result outer-shape preservation, \$regex rejection, registry lookup,
priority resolver, and PalaceRef-kwarg ChromaBackend.get_collection.

Refs: #743 (RFC 001), #989 (RFC 002 tracking issue).
jphein added a commit to jphein/mempalace that referenced this pull request Apr 19, 2026
Scanned all 233 open upstream PRs today against our open PRs and
fork-ahead / planned-work items. Findings merged into README:

- P2 (decay) and P3 Tier-0 (LLM rerank): both covered by MemPalace#1032
  (@zackchiutw, MERGEABLE, 2026-04-19 — Weibull decay + 4-stage
  rerank pipeline). Older simpler version at MemPalace#337. Dropped as
  fork work; watching MemPalace#1032.
- P7 (alternative storage): formally out of scope. RFC 001 MemPalace#743
  (@igorls) defines the plugin contract; four backend PRs already
  in flight (MemPalace#700, MemPalace#381 Qdrant; MemPalace#574, MemPalace#575 LanceDB). Fork consumes,
  does not rebuild.
- P0 (multi-label tags): still fork/upstream candidate. MemPalace#1033
  (@zackchiutw) ships adjacent privacy-tag + progressive disclosure
  but not the full multi-label scheme.
- Merged MemPalace#1023 section acknowledges complementary MemPalace#976 (felipetruman)
  which adds broader mine_global_lock() + HNSW num_threads pin.

Gives future-us a map so we don't re-file MemPalace#1036-style duplicates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants