Good call on the shared utility — went ahead and did the refactor. Extracted the pagination loop into palace.py as iter_metadatas(col, where=None, batch=500), alongside get_collection and file_already_mined (that module's own docstring already calls it out as the home for shared ChromaDB patterns). miner.status() now just iterates iter_metadatas(col) — the inline loop is gone. mcp_server.py can swap its own _iter_all_metadatas for the shared one once #307 merges.

The MCP tools are covered — #493 fixes all four (tool_status, tool_list_wings, tool_list_rooms, tool_get_taxonomy) using the same paginated offset pattern. This PR stays scoped to the CLI/miner path.

Got it — good to know #493 covers the MCP side. The split makes sense: fixing the CLI path here and the MCP tools in a focused PR there is cleaner than trying to do it all in one. Both can land independently.

Scope has expanded since that comment. After the discussion on #478, went ahead and did the full consolidation: removed _iter_metadatas from mcp_server.py and the inline loop from palace_graph.py — both now use iter_metadatas() from palace.py. Single implementation across the whole codebase.

That's a cleaner outcome than the scoped fix — single iter_metadatas() implementation across the whole codebase means there's exactly one place to update when ChromaDB changes its pagination API again (and it will). Good call on expanding scope after the #478 discussion.

Does this now supersede the pagination work in PR #563 (the col.count() > 10000 dynamic limit fix), or are they complementary? Asking because both seem to be addressing the 10K wall from different angles — #563 fixes the limit query, this PR fixes the shared implementation. Might be worth coordinating to avoid merge conflicts.

Looked at #563 — they conflict rather than complement. #563 replaces limit=10000 with limit=max(col.count(), 1) in mcp_server.py, which is a single unbounded fetch that scales with palace size. Our PR already removed those lines in favour of iter_metadatas(). Beyond the merge conflict, the max(col.count(), 1) approach trades one problem for another — on a 125k-drawer palace it fetches all 125k records in a single col.get() call, which is the kind of memory spike the pagination is meant to avoid.

Got it — #482 already removed those lines in favour of iter_metadatas(), so #563's max(col.count(), 1) patch is patching code that doesn't exist in this branch. The pagination approach is strictly better for large palaces anyway; the single-fetch fix just moves the memory wall up rather than removing it.

@milla-jovovich @bensig — rebased and conflict-free. Quick summary of where this stands:

• Fixes the 10k truncation bug in miner.status() (CLI path of MCP server: mempalace_list_wings and mempalace_get_taxonomy return empty despite data in palace #171)
• Extracts iter_metadatas() into palace.py as a shared utility, consolidating the separate pagination loops in mcp_server.py and palace_graph.py onto a single implementation
• jphein has confirmed the pagination piece from the feat: batch writes, concurrent mining, MCP tools, hooks, export, search improvements #562 split will defer to this PR rather than duplicate it (fix: replace hardcoded limit=10000 with dynamic col.count() in navigation tools #563 is also effectively superseded)
• 535 tests passing; the 3 pre-existing failures in TestTaxonomyPagination are from a test helper signature mismatch already present on main

Rebased again onto upstream/main after PR #371 landed — that PR added inline pagination loops to mcp_server.py which conflicted with our iter_metadatas() consolidation. Resolved in favour of the shared utility, incorporating the "partial": true response key from #371's tool_status implementation. Conflicts clear, 590 tests passing.

Closing — this is superseded by recently merged PRs to develop. Thank you for the contribution!