Skip to content

mempalace status crashes with "too many SQL variables" on large palaces #802

@srgvg

Description

@srgvg

Bug

mempalace status crashes with:

sqlite3.OperationalError: too many SQL variables

Root cause

miner.py calls col.get(limit=total, include=["metadatas"]) without pagination. SQLite binds one host variable per metadata item, hitting the compile-time limit (~32766) when the palace exceeds ~30k drawers.

Reproduction threshold

Any palace with more than ~30k drawers. Confirmed on a palace with 50k+ drawers.

Workaround

sqlite3 ~/.mempalace/palace/chroma.sqlite3 "SELECT COUNT(*) FROM embeddings;"

mempalace search continues to work correctly.

Fix

Paginate the col.get() call in batches of 5000. ChromaDB's get() accepts an offset parameter, so this is straightforward.

See PR from srgvg/mempalace for the one-function fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/cliCLI commandsbugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions