Skip to content

Interrupted mine corrupts persisted palace and causes segfault #72

@ac-opensource

Description

@ac-opensource

What happened?

I mined a large Rust project with mempalace. During indexing, the machine became extremely slow, then froze, and then restarted automatically, apparently under heavy RAM and storage pressure.

After reboot, the existing palace became unreadable through mempalace normal status path:

  • mempalace status crashes with a segmentation fault
  • opening the Chroma client works
  • listing collections works
  • reading the existing collection with count() or get() segfaults

What I found:

  • the crash is inside Chroma's Rust-backed local read path, not in mempalace CLI startup
  • a fresh temporary Chroma database works correctly on the same machine
  • the existing palace's SQLite metadata is still present and readable
  • the stored drawer text is still present in SQLite
  • the failure is specific to the existing persisted Chroma index/state after the crash

This looks like the forced restart during mining left the persisted Chroma local index in a corrupted or inconsistent state. mempalace currently does not handle that case gracefully, and the read path segfaults instead of returning a recoverable error.

Observed stack location with faulthandler:

Fatal Python error: Segmentation fault

Thread 0x00000002054ab100 (most recent call first):
  File ".../site-packages/chromadb/api/rust.py", line 421 in _get
  File ".../site-packages/chromadb/api/models/Collection.py", line 159 in get

I also reproduced a similar segfault through count():

File ".../site-packages/chromadb/api/rust.py", line 378 in _count

Palace size when this happened:

  • palace size: about 295 MB
  • chroma.sqlite3: about 224 MB
  • vector index files: about 70 MB
  • embeddings count: 42,003

What did you expect?

Mining should either:

  • complete successfully, or
  • fail safely without destabilizing the machine or corrupting the palace

After an interrupted run or system restart, mempalace status should either:

  • still work, or
  • return a recoverable error / repair path

It should not segfault.

How to reproduce

  1. Run mempalace mine <large-rust-project>.
  2. If the machine crashes/restarts under memory or storage pressure during or after mining, reboot.
  3. Run mempalace status.
  4. Observe a segmentation fault.

Environment

  • OS: macOS 26.3.1
  • Python version: 3.9.6
  • MemPalace version: 3.0.0
  • ChromaDB version: 1.5.6

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions