feat: mempalace migrate — recover palaces from different ChromaDB versions#502
feat: mempalace migrate — recover palaces from different ChromaDB versions#502
Conversation
…abels - Add AGENTS.md with build commands, project structure, conventions - Add .github/dependabot.yml for automated pip + actions updates - Add .github/CODEOWNERS for review routing - Expand .gitignore (.env, .DS_Store, IDE configs, coverage, venvs) - Add C901 complexity rule to ruff (max-complexity=25, benchmarks excluded) - Add --durations=10 to pytest CI for test performance tracking - Add docs/schema.sql for knowledge graph schema documentation - Created P0-P3 priority + area/* + security/performance/docs labels
…sions Reads documents and metadata directly from ChromaDB's SQLite (bypassing the API that fails on version-mismatched databases), then reimports into a fresh palace using the currently installed ChromaDB. Fixes the 3.0.0 → 3.1.0 upgrade path where chromadb was downgraded from 1.5.x to 0.6.x, breaking the on-disk storage format. - Detects chromadb version from SQLite schema (0.6.x vs 1.x) - Extracts all drawers with full metadata via raw SQL - Builds fresh palace in temp dir, swaps atomically - Backs up original palace before any changes - Supports --dry-run to preview without modifying Fixes #457
web3guru888
left a comment
There was a problem hiding this comment.
The SQLite-bypass approach here is exactly right — it's the only reliable path when the ChromaDB API itself is broken. A few observations from our integration:
migrate.py looks solid. The raw SQL extraction handles the 1.x → 0.6.x schema gap cleanly, the backup-then-swap pattern is safe, and the dry-run flag is genuinely useful (we would have used this during our own upgrade pain). The version detection via schema_str column presence is the right heuristic.
One gap worth addressing: extract_drawers_from_sqlite queries on embedding_id but the inner metadata query uses e.id (embeddings.id) not embeddings.embedding_id. In most cases these map correctly through the JOIN, but the two-query structure leaves a subtle mismatch possibility: if rows returned for the same embedding_id have different e.id values (e.g. duplicate entries from a failed partial write), the metadata merge could silently drop keys. Consider joining on embedding_id throughout or at minimum asserting uniqueness.
Embeddings not migrated — explicitly noted in the PR, but worth adding a user-visible line in the migrate output: something like 'Embeddings will be regenerated on first search — initial queries after migration may be slow.' Users who haven't read the code will wonder why the first search takes 10x longer.
CODEOWNERS concentration — same note as in #497: bensig now owns .claude-plugin/, .codex-plugin/, integrations/, .github/ and is co-owner of mempalace/. That's a single-point bottleneck if unavailable. Fine for now but worth a comment in CODEOWNERS.
The --durations=10 CI addition is useful — we've caught slow tests that way. The C901 complexity rule is a good addition too.
534 tests passing on a PR with a new migrate.py is a bit surprising — are there dedicated tests for extract_drawers_from_sqlite and detect_chromadb_version? The coverage count may be coming from the other files in this batch rather than the new migration logic. Worth adding at minimum a unit test with a mock SQLite db that matches the 1.x and 0.6.x schema patterns.
web3guru888
left a comment
There was a problem hiding this comment.
This is a solid approach. Reading directly from SQLite to bypass a broken ChromaDB API is exactly the right call — it's the only reliable way to recover data across major schema changes, and it's robust because SQLite's own format hasn't changed.
A few observations from reading the implementation:
The raw SQL query works but has a gap:
SELECT e.embedding_id,
MAX(CASE WHEN em.key = 'chroma:document' THEN em.string_value END) as document
FROM embeddings e
JOIN embedding_metadata em ON em.id = e.id
GROUP BY e.embedding_idThis assumes ChromaDB 1.x stores the document text as chroma:document in embedding_metadata. If a palace was created with an older 0.4.x schema (some users upgraded from 3.0.0 which shipped with >=0.4.0), the document storage key may differ. Worth testing against a real 0.4.x-created palace before merging, or adding a fallback query path.
Batch import uses col.add(), not col.upsert(): If the migration is re-run (e.g., interrupted and restarted), the second run will crash on the first batch containing an ID that's already in the fresh palace. Consider col.upsert() or a check-before-import guard to make re-runs safe. This matters especially because a power failure between swap and completion would leave users stranded.
Connection leak in extract_drawers_from_sqlite: The conn.close() call at the end is correct, but if either SQL query raises (malformed DB, unexpected schema), conn leaks. A context manager (with sqlite3.connect(db_path) as conn) would close it on error too. The detect_chromadb_version function already uses finally: conn.close() correctly — same pattern here.
Missing: KG migration. The SQLite knowledge graph at ~/.mempalace/knowledge_graph.db is not ChromaDB-dependent and doesn't need version migration, but it's worth noting in the output that KG data is preserved separately. Users who've been burned by the 3.0→3.1 upgrade might not realize their KG is safe.
# Could add after migration completes:
kg_path = os.path.expanduser("~/.mempalace/knowledge_graph.db")
if os.path.exists(kg_path):
print(f" Note: Knowledge graph at {kg_path} is unaffected.")The dry-run, atomic backup-then-swap, and version detection logic are all well-designed. This fills a real gap — we've seen multiple reports in #457 and #469 of users losing palace data on upgrade. With the upsert fix and connection guard, this is close to ready.
|
confirmed from #469 it works for my old palace data. |
|
This is exactly the tool the community has been missing. The ChromaDB version break has been one of the most painful issues we've dealt with in our integration — we ended up pinning The implementation looks solid. A few things worth testing if you haven't already: Idempotency: if someone runs Interruption recovery: if the process is killed mid-swap (between backup and move), what's the recovery path? The atomic swap approach is good, but it'd be reassuring to document what state the user is left in and how to resume — even just a note in the Neither of these feels like a blocker given R0uter's confirmation and the manual test coverage. Just good edge cases to have in the test plan before merge. Nice work on this one. |
milla-jovovich
left a comment
There was a problem hiding this comment.
Looks good, merge it
Summary
Adds
mempalace migratecommand that recovers palaces created with a different ChromaDB version. Reads documents and metadata directly from SQLite (bypassing the ChromaDB API that fails on version-mismatched databases), then reimports into a fresh palace.Fixes the 3.0.0 → 3.1.0 upgrade path where chromadb was silently downgraded from 1.5.x to 0.6.x, breaking the on-disk format.
How it works
schema_strcolumn = 1.x)Usage
Tested
mempalace migrate— all drawers recovered with full metadatastatusandsearchwork on migrated palaceFixes #457
Test plan
ruff check ./ruff format --check .— cleanpytest tests/ -v— 534 passed