Summary

The capacity probe added in #1227 hardcoded a 2,000-row floor for the "DIVERGED" decision. The comment justifying that number explicitly tied it to chromadb's default sync_threshold of 1,000:

Two synchronization windows worth (2 × sync_threshold = 2000) is a safe steady-state ceiling

#1191 then set hnsw:sync_threshold = 50_000 via _HNSW_BLOAT_GUARD (to prevent index bloat) without updating the divergence floor to track. The two changes shipped about a week apart and the dependency wasn't caught.

Symptom

Any palace created via ChromaBackend.create_collection after #1191 carries hnsw:sync_threshold=50_000 in collection_metadata. ChromaDB then naturally accumulates up to 50,000 entries in its WAL (embeddings_queue) between HNSW flushes — that's by design. The on-disk index_metadata.pickle only updates at flush time, so between flushes:

• sqlite_count grows with every write
• hnsw_count (read from the pickle) stays frozen until the next flush
• Divergence climbs from 0 → ~50K → flush → back to 0 → repeat

With the 2,000 floor, the divergence threshold becomes max(2000, 10% × sqlite_count). For a 100K-drawer palace that's 10,000 — and the queue crosses 10K maybe a fifth of the way through every 50K-write cycle. Result: the MCP server (mcp_server._refresh_vector_disabled_flag) routes vector search to the BM25 fallback and disables duplicate detection for ~80% of the write cycle on any actively-mined ≥100K palace, even though chromadb is behaving correctly.

Concretely on a 100K-drawer palace I just rebuilt:

$ mempalace repair-status
  [drawers]
    sqlite count:   98,176
    hnsw count:     50,000
    divergence:     48,176
    status:         DIVERGED
    note:           HNSW index holds 50,000 elements but sqlite has 98,176
                    embeddings — 48,176 drawers (49%) are invisible to vector
                    search. Run `mempalace repair` to rebuild.

The "49% invisible" framing is misleading: those 48,176 drawers are queued for the next HNSW flush, not corrupt. ChromaDB's WAL replay surfaces them on every fresh open — count() returns the full 98,176, and a CLI mempalace search returns hybrid (cosine + BM25) results across the full set. The MCP server's stricter guardrail is what disables vector search.

Fix

Read the configured hnsw:sync_threshold from collection_metadata per palace and scale the floor to 2 × sync_threshold. The 10% relative term, the unflushed-pickle branch, and the original #1222 detection capability are unchanged.

sync_threshold = _read_sync_threshold(palace_path, collection_name)
divergence_floor = max(_HNSW_DIVERGENCE_FALLBACK_FLOOR, 2 * sync_threshold)
threshold = max(divergence_floor, int(sqlite_count * _HNSW_DIVERGENCE_FRACTION))

A 91%-missing-of-192,997 palace (the actual #1222 reproducer) still trips, regardless of whether the collection was created with sync_threshold=1000 or 50000.

Behavior summary

Tests

Four new tests in tests/test_hnsw_capacity.py:

• test_capacity_status_tolerates_lag_under_large_sync_threshold — regression for the fix: prevent HNSW index bloat from resize+persist cycles #1191/fix(repair): detect HNSW capacity divergence and fall back to BM25 (#1222) #1227 conflict: 100K sqlite + 50K HNSW + sync_threshold=50_000 → OK. (Pre-fix: would flag DIVERGED.)
• test_capacity_status_still_flags_real_corruption_under_large_sync — HNSW max_elements frozen at 16K while collection grows to 100K+ entries — MCP server segfaults on every tool call #1222-shape (200K sqlite + 16K HNSW) on a sync_threshold=50_000 collection still trips correctly.
• test_capacity_status_default_threshold_when_no_sync_metadata — legacy palaces without the metadata row use the 2,000 fallback floor (no behavior change).
• test_unflushed_path_also_uses_dynamic_floor — the never-flushed branch (no pickle yet) also scales: a 30K-drawer collection under sync_threshold=50_000 is no longer flagged DIVERGED before its first flush.

_seed_chroma_db in the test fixture grew an optional sync_threshold parameter that, when provided, seeds an hnsw:sync_threshold row into a new collection_metadata table. All 18 pre-existing tests in test_hnsw_capacity.py continue to pass; the broader test_backends.py (45 tests) shows no regressions.

Related

• fix: prevent HNSW index bloat from resize+persist cycles #1191 — the PR that set _HNSW_BLOAT_GUARD with sync_threshold=50_000
• fix(repair): detect HNSW capacity divergence and fall back to BM25 (#1222) #1227 — the PR that added the capacity divergence detection (and the hardcoded 2,000 floor this PR makes dynamic)
• HNSW max_elements frozen at 16K while collection grows to 100K+ entries — MCP server segfaults on every tool call #1222 — the original failure mode the divergence detection was added to catch (still detected after this fix)