Repository navigation
fix(setup): persist backend marker before setup.py can fail; close ML-child isolation gap - #638
Conversation
…-child isolation gap Two Windows-reported defects (issue #633), both cross-platform in the underlying pin lists / code paths even though only observed there: 1. install-plugin.sh wrote ~/.claude/methodology/backend.json only after running scripts/setup.py, guarded by `|| fail`. Any setup failure discarded the already-correct Phase 0 backend decision. On the next launch, memory_config._detect_runtime() defaults to "cli" (the plugin's own CORTEX_RUNTIME="" resolves there), and memory_store._construct_store() forces backend="postgresql" whenever runtime=="cli" and backend=="auto", so a missing marker silently converted a chosen SQLite install into a hard PostgreSQL requirement, with none ever provisioned. Fixed: the marker now persists immediately after the backend is decided, before setup can fail and exit. 2. scripts/setup.py's cache_embedding_model() and its macOS/Linux twin scripts/lib/precache_embedding_model.sh spawn a child process that reaches deps/ only through PYTHONPATH, which cannot cut user site-packages the way launcher_site.isolate_deps() does in-process (issue #621), so a conflicting user-site torch/torchaudio build could still abort the embedding-model pre-cache. Both now bootstrap isolate_deps() in the child before importing sentence_transformers. scripts/setup.py's cache_embedding_model payload is pulled into a standalone, unit-tested _model_cache_child_source() function. _model_checks() also widens ImportError to Exception, matching _postgres_checks()'s own pattern, so a torch ABI mismatch (OSError or RuntimeError) reports a [FAIL] row instead of crashing verify() outright, closing the "exited before reporting any check" failure mode. setup_database() needed no such bootstrap (its child imports only psycopg and pg_schema, not the ML stack); audited and documented rather than assumed. Decisions recorded as ADR-1087 and ADR-1088 (wiki/adr/), per the decision-gate: large rationale blocks belong in the wiki, code keeps a short `# source: ADR-NNNN` pointer. New regression test (test_install_plugin_marker_order.py) drives the real installer against a python3 stub that fails only scripts/setup.py, proving the marker survives, verified to fail on the pre-fix script and pass on the fixed one. New tests compile and run _model_cache_child_source() end-to-end against a stubbed sentence_transformers, and prove repr() escapes Windows-shaped paths (backslashes, an apostrophe) correctly. Full suite: 9117 passed, 11 skipped (darwin). Windows unverified in this environment; CI's ci-sqlite-min.txt job is the nearest proxy. Known: this branch's ADR-1087/1088 and the sibling issue #634 fix branch's ADR-1087 were assigned independently from the same tip and collide on number 1087 with different content. Whichever PR merges second must renumber (file, wiki/manifest.json entry, mirror under docs/adr/, and the `source: ADR-1087` pointers in this branch's scripts/setup.py and scripts/lib/precache_embedding_model.sh). Co-Authored-By: Claude Sonnet 5 <[email protected]>
Reviewed and reworked. The root cause the previous commit on this branch (715f580) never touched: SqliteMemoryStore._insert_memory_rows, reembed_memory, and update_memory_compression each stamp memories.embedding_model unconditionally whenever an embedding is present, whether or not the vec INSERT for it actually succeeds. _stamp_embedding_model's own docstring already stated the precondition every caller violated ("memory_id refers to a just-written vec row"). A memory written while sqlite-vec is unavailable -- the exact population issue #634 describes -- gets stamped 'neural' with no row in memories_vec behind it. The tag has two readings tangled into one write path. 'fallback' records which encoder ran; the re-embed worklist keys on it (WHERE embedding_model = 'fallback'), and a fallback memory with no vector must still surface for upgrade, so it stamps unconditionally -- issue #169's own tested contract (test_embedding_model_stamped_fallback) requires exactly this and is unchanged. 'neural' is read elsewhere (sqlite_store_search.py's mixed-store cross-rank filter) as a claim that a neural vector exists; stamping it with no vector behind it is issue #634's bug, in one word: 'neural' stamped without a vector. _insert_memory_rows and update_memory_compression now resolve the model name and whether the vec write actually succeeded, then stamp when model != "neural" or persisted: 'fallback' always stamps, 'neural' stamps only when memories_vec genuinely received the row. reembed_memory needed no such split -- its only caller, run_embedding_upgrade_cycle, now checks store.has_vec once before calling it at all, so a persisted vector is already guaranteed by the time it runs. run_embedding_upgrade_cycle gains has_vec on the _FallbackWorklistStore Protocol and returns {"upgraded": 0, "reason": ...} before encoding or restamping anything when has_vec is false, touching nothing -- the correct behavior for a cycle that cannot persist what it would compute, not a reason to restamp and report the gap after the fact. The vectors_persisted field the previous commit added to paper over that restamping is deleted; there is nothing left for it to report. select_fallback_embeddings's widened worklist query (the NOT IN memories_vec sweep) is kept: once the stamp fix lands, no production write path can produce a 'neural'-without-vector row going forward, so the sweep's remaining job is a one-time migration, self-healing installs whose database already carries rows the pre-fix bug wrote (the reporter's own measured 86 such rows). _sqlite_vector_search (mcp_server/doctor.py) changes from optional=True to optional=False: sqlite-vec now ships in every install's base package set, so its absence or a load failure is a broken install, not a supported degraded mode. Craftsmanship gate follow-up: this branch's first commit pushed mcp_server/doctor.py, embedding_upgrade.py, and tests_py/test_doctor.py over the method-size and file-size caps with inline rationale that belonged in the wiki ADR, not the code -- caught by this branch's own CI run once triggered. Docstrings trimmed to short pointers; the sqlite-vec-specific tests move to a new tests_py/test_doctor_sqlite_vector_search.py to keep test_doctor.py under 300 lines. tests_py/invariants/test_I2_canonical_writer.py's ALLOWED_WRITERS line numbers for sqlite_store.py shifted by +3 from this diff; updated to match (same three writers, confirmed by reading each site, not assumed). ADR renumbered 1087 -> 1089 (this branch's own number; sibling PRs #638 and #639 already claim 1088/1090) and rewritten around the actual defect ('neural' stamped without its vector) instead of only the install-list gap; the wiki ADR is now this fix's single source of truth for the 'fallback' vs 'neural' semantics, referenced by every call site that must not re-collapse them. Left unaddressed, flagged for the maintainer: _try_load_vec logs INFO and sets has_vec=False when the extension fails to load. Hardening it to raise at store-open time would change startup behavior on interpreters without loadable-extension support (common on macOS system/Framework Python) -- a product decision, not bundled here. Full suite: 9121 passed, 11 skipped (darwin, same pre-existing 11). ruff and the craftsmanship gate both clean. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
…-marker # Conflicts: # CHANGELOG.md
…late Caught by this branch's own craftsmanship gate once a real CI run finally ran on it: test_marker_persists_backend_on_a_successful_sqlite_install exceeded the 40-line method cap, duplicating the stub-writing and env-building steps already in the sibling failure-path test almost verbatim. Extracted _write_setup_py_stub(bin_dir, exit_code=...) (the prior failure-only stub, now parametrized on the stubbed exit code) and _run_installer(tmp_path, setup_py_exit_code=...) (the shared home/PATH-isolated subprocess.run call), plus a small _marker_backend() reader both tests' final assertions were duplicating too. Both tests now read as their actual distinguishing behavior (exit code in, marker survives or not) instead of repeating the installer-driving mechanics. No behavior change: both tests still pass unchanged. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
install-plugin.sh's stale-install pruning (phase 2b) runs pip3 show/ uninstall unconditionally against whatever pip3 resolves to on PATH. This test only stubbed python3, so in CI pip3 resolved to the real interpreter's pip and genuinely uninstalled this repo's own just-installed editable hypermnesia-mcp package, breaking the "Verify hook-free MCP host contract" step later in the same job (FileNotFoundError: hypermnesia-mcp). Reproduced deterministically via a CI rerun of PR #638 (same two jobs failed identically both times). Stub pip/pip3 as always-report-not-installed, matching the existing python3 stub pattern, so the pruning phase's uninstall branch is never reached. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
|
ZETETIC-REVIEW: REQUEST_CHANGES Stakes: High (install/launch-path correctness gate for every plugin user; touches the sole determinant of which storage backend a launch resolves to). What holds up. The primary fix is correctly implemented: I traced both branches in scripts/install-plugin.sh (sqlite via run_setup_py, postgres via setup.py-on-Windows or setup.sh-on-Darwin/Linux) and the marker write now sits before every fail-and-exit path in all three. The except Exception widening in scripts/setup.py::_model_checks is scoped tightly, one import statement per try block, matching the existing _postgres_checks pattern exactly — it does not swallow anything outside the two named optional-dependency probes. The pip/pip3 stub fix in tests_py/scripts/test_install_plugin_marker_order.py is sound: it makes Two problems in the diff itself.
Non-blocking: Everything else — layering, SRP/OCP on the extracted _model_cache_child_source, wiring, ADR pointers, craftsmanship-gate size limits — is clean. |
Review (PR #638) caught a real regression: persisting the marker before setup runs (this PR's own #633 fix) unconditionally overwrote a previously-persisted, working backend choice with the newly requested one before that new backend was ever confirmed working. A user switching from a working sqlite install to --postgres, whose postgres setup then fails, ended up with the marker pointing at postgresql with nothing provisioned -- issue #633's exact bug class, in reverse. Capture the marker's previous backend before overwriting it, and restore it whenever the newly requested backend differs and its setup fails. A fresh install (no prior marker) or a same-backend reinstall have nothing to restore, so #633's original scenario is untouched. Added a regression test that seeds a working sqlite marker, requests a postgres switch through the Windows branch (uname stubbed, which reuses the same stubbable scripts/setup.py path already used for the sqlite case), fails setup, and asserts the marker still reads sqlite. Also fixed: scripts/lib/precache_embedding_model.sh interpolated project_dir/deps_dir directly into single-quoted Python string literals, unlike its Python twin (_model_cache_child_source), which escapes via repr() per ADR-1087. A path containing a single quote or backslash would break the embedded Python source. Pass both paths through env vars instead, which need no escaping at all. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
|
ZETETIC-REVIEW: APPROVE Both blocking findings from the previous review are resolved in c43eecb. Finding 1 (marker-write regression on backend switch), resolved. Verified by mutation: I checked out the pre-fix commit (fd7499e) and ran the new test Finding 2 (bash string-interpolation escaping), resolved. Verified by mutation: I ran both the old and new CI is green across the board: Lint, Craftsmanship Gate, Type Check, Test 3.10 through 3.13, Windows SQLite, Docker Smoke, Fuzz. Non-blocking follow-ups (do not block merge):
|
…ll (#639) * fix(consolidate): turn silent PG-only crashes into named skips on SQLite Windows-reported (issue #636), but the root cause is not platform-specific: consolidate's wiki maintenance cycle runs four passes on every invocation (wiki_source_backfill_pass, wiki_domain_backfill_pass, wiki_citation_seed_pass, wiki_backlog_pass's lesson-promotion count), each needing store.batch_pool (a psycopg_pool.ConnectionPool only PgMemoryStore provides, since the SQL is Postgres-schema-qualified: wiki.pages, wiki.citations, %s placeholders). SqliteMemoryStore has no batch_pool attribute at all, so every one of these passes raised AttributeError on every SQLite install, on every run. Three of the four caught that AttributeError internally and returned {"status": "error: ..."} instead of letting it propagate. wiki_maintenance.py's own escalation only fired on a RAISE past the await, never on a returned error stanza, so consolidate kept reporting status: "ok" and failed_stages: [] while four maintenance passes were silently broken on the default backend, on every run. The fourth pass already degraded to None on the same AttributeError but mischaracterized SQLite as "fallback in tests", and called silent_failure.note() on every occurrence, incrementing cortex_silent_failures_total on every consolidate run for the lifetime of a SQLite install for a gap that isn't actually a failure. Fixed: - Added mcp_server/handlers/consolidation/batch_pool_capability.py, a shared batch_pool_skip_reason(store) guard. Every batch_pool-gated pass now checks it before entering its try block and returns {"status": "skipped: store has no batch_pool (non-PostgreSQL backend)"} instead of ever reaching AttributeError. Applied to the three consolidate-wired wiki passes and, since audit confirmed the identical unconditional store.batch_pool pattern, to the five standalone-campaign passes the issue flagged as unverified suspects (memory_dedup_exact_pass, memory_domain_backfill_pass, memory_reheat_pass, near_dup_calibration_pass's two entry points, write_class_backfill_pass). - wiki_maintenance.py gained _escalate_if_error(): it now also escalates out["status"] when a sub-pass RETURNS (not just raises) an "error:" status, fixing the previously dead escalation path. A "skipped: ..." status does not escalate. - wiki_backlog_pass._lesson_promotion_backlog checks the same guard first and returns None without calling silent_failure.note() when the capability is simply absent, reserving that call for a genuine query failure once batch_pool exists. Docstring no longer calls SQLite a test-only fallback. Decision recorded as ADR-1089 (wiki/adr/), per the decision-gate. Not 1087: two sibling branches from this same reporting session (fix/issue-634-sqlite-vec, fix/issue-633-backend-marker; PRs #637, #638) already claimed 1087 and 1088 independently from the same main tip, and both document that collision against each other. 1089 avoids adding a third branch to it; whichever PR merges last still renumbers as needed. Left out of scope, flagged for a separate decision: ingest_codebase.py's entity/edge ingestion also references store.batch_pool unconditionally, with no try/except at all, but its sink SQL uses PostgreSQL's COPY ... FROM STDIN wire protocol (mcp_server/infrastructure/staging_resolve_sink.py), which has no SQLite equivalent. This is a materially larger, pre-existing, PG-only architectural gap (crashes loudly today, not silently), not a simple missing-guard bug. New tests: unit coverage for the guard helper; a contract test running all nine fixed passes against a real, fresh SqliteMemoryStore built through the composition root, asserting every one degrades to "skipped: ..." and never "error: ..."; wiki_maintenance tests proving a returned (non-raised) error stanza now escalates, and that a skipped stanza does not. Full suite: 9157 passed, 11 skipped (darwin). Windows unverified in this environment. Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(consolidate): gate PG-only wiki passes once at wiring, not per call Reworks the previous commit (efac006) on this branch. That commit fixed issue #636's reported symptom (consolidate reports status ok while four wiki passes silently fail on SQLite) by adding a hasattr(store, "batch_pool") guard inside each of nine call sites, turning the AttributeError into a returned {"status": "skipped: ..."} stanza. Reviewed and rejected: that is a catch-and-degrade shim restating the same defect with a softer word, not a fix at the root, and the repo's own non-negotiable already says so ("no silent fallbacks or compatibility shims"). This commit fixes the actual root cause instead: wiki_maintenance.py wired four PostgreSQL-only passes (wiki_source_backfill_pass, wiki_domain_backfill_pass, wiki_citation_seed_pass, wiki_backlog_pass's lesson-promotion count) unconditionally, regardless of backend. run_wiki_maintenance now decides once, at the top, whether store is PostgreSQL-backed (hasattr(store, "batch_pool"), the same idiom this file already used for iter_memories_for_decay) and only invokes those four inside that branch. On SQLite they are simply absent from the response, the same way "wiki" itself is absent from consolidate's stats when wiki=False -- not reported as failed, not reported as skipped. The four passes no longer catch their own AttributeError (or any exception) for a missing-capability case: run_wiki_maintenance is each one's sole caller now, so being invoked without batch_pool is a wiring bug and raises, like any other broken precondition. This also revives wiki_maintenance's existing except blocks around each await as their real, only error boundary -- previously dead code, since every pass swallowed its own exception before the caller ever saw it. wiki_backlog_pass._lesson_promotion_backlog moves out of run_backlog_pass (unconditional, filesystem-based) into run_wiki_maintenance's own PostgreSQL branch; its try/except and silent_failure.note for a genuine query failure are unchanged (ADR-0640's accepted pattern), only the call site moved, which is what stops the metric from incrementing on every SQLite consolidate run. The five standalone campaign scripts sharing the identical store.batch_pool pattern (memory_dedup_exact_pass, memory_domain_backfill_pass, memory_reheat_pass, near_dup_calibration_pass's two entry points, write_class_backfill_pass) are not wired into consolidate and keep their original, pre-#636 try/except (a human runs the matching scripts/*.py CLI directly and reads the printed status) unchanged. Each script's entry point now checks hasattr(store, "batch_pool") once, before calling its pass, and exits 1 with a one-line message when absent. Deleted batch_pool_capability.py and its unit test (the shim and its own test); deleted test_wiki_batch_pool_sqlite_contract.py (tested the shim's "skipped:" status shape, no longer applicable). New tests in test_wiki_maintenance.py: the four PG-only keys are absent on a store without batch_pool and present on one with it; a contract test runs the whole cycle against a real, fresh SqliteMemoryStore built through the composition root and asserts no stanza anywhere has an "error:" status and the four keys are absent -- issue #636's own requested check. consolidate's tool schema documents the four keys as PostgreSQL-only. ADR renumbered 1089 -> 1090 (this branch's own number; the sibling PRs #637/#638 already claim 1087/1088) and rewritten to describe this design, including why the first (shim) version was rejected on review. Left out of scope, flagged separately: ingest_codebase.py's entity/edge ingestion also references store.batch_pool unconditionally, but its sink SQL uses PostgreSQL's COPY ... FROM STDIN wire protocol with no SQLite equivalent -- a materially larger, pre-existing, architectural gap, not a missing-guard bug. Full suite: 9146 passed, 11 skipped (darwin, same pre-existing 11 as before this branch). ruff check/format and the craftsmanship gate both clean. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik * fix(consolidate): escalate lesson_promotion_backlog failures like the others Advisor review of the previous commit (f9398f1) on this branch found one remaining inconsistency: _lesson_promotion_backlog kept its own internal try/except, so a genuine PostgreSQL query failure (pool exhausted, table missing) degraded to lesson_promotion_backlog: None while out["status"] stayed "ok" -- the exact shape issue #636 reported, just for one key instead of four, and the one PG-only pass this branch's rework had not yet made consistent with the other three. Removed the try/except from _lesson_promotion_backlog: it now raises on a query failure like source_backfill/domain_backfill/citation_seed do. run_wiki_maintenance wraps the call in its own try/except and escalates out["status"] to "lesson_promotion_backlog_error: ..." on failure, matching the other three exactly. This also drops the silent_failure.note call the function no longer needs -- the wiring fix from the previous commit already means it is never invoked at all on a non-PostgreSQL store, which is what actually stops the metric pollution, not a swallowed exception. Regenerated .craftsmanship-baseline.json: removing wiki_backlog_pass.py's now-unused `mcp_server.observability` import fixed a pre-existing layer-violation baseline entry, which needed pruning as stale. New tests: wiki_backlog_pass's own unit tests now assert raise instead of a degraded None; a new wiki_maintenance test proves a real failure there escalates status exactly like the citation-seed equivalent. Full suite: 9147 passed, 11 skipped (darwin, same pre-existing 11). ruff and the craftsmanship gate both clean. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik * fix(tests): build the contract test's store through the real composition root Review (PR #639) caught the contract test (TestRealSqliteStoreContract, issue #636's own requested check) instantiating SqliteMemoryStore(path) directly, bypassing get_shared_store/MemoryStore/_construct_store -- exactly the backend-selection logic this bug is about. Rebuilt it to force CORTEX_MEMORY_STORE_BACKEND=sqlite and go through get_shared_store(), clearing the memory-settings cache and shared-store singleton before and after, matching the pattern already used in tests_py/hooks/test_entry_backend_parity.py. Also fixed the non-blocking duplication finding: the five campaign scripts (memory_dedup_exact, memory_domain_backfill, memory_reheat, near_dup_calibrate, backfill_write_class) each carried an identical 7-line batch_pool guard. Extracted to mcp_server/shared/campaign_store_guard.py::require_postgres_store, behavior-preserving. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik --------- Co-authored-by: Claude Sonnet 5 <[email protected]>
…s rows (#637) * fix(sqlite): install sqlite-vec on every backend, self-heal vectorless rows Windows-reported (issue #634), but the root cause is not platform-specific: neither dependency-install path the plugin uses ever named the [sqlite] extra. - scripts/launcher_pins.py::BASE_PACKAGES (installed by ensure_deps() on every hook/session-start launch) listed the PostgreSQL drivers unconditionally but nothing from [sqlite]. - requirements/setup.txt, generated for extras (postgresql, codebase, benchmarks), is what scripts/setup.py::install_deps() installs on BOTH backends (SKIP_POSTGRES does not change which file is read). Consequence: the SQLite backend, the zero-config default, ran with vector search silently disabled (FTS-only) on every install, on every OS, while check_setup/doctor reported the store fully ready. Fixed: - Added "sqlite" to setup.txt's generated extras and sqlite-vec==0.1.9 to BASE_PACKAGES, pinned to match. Existing installs self-heal on next launch (ensure_deps()'s pin-satisfaction check compares sorted(pins), so the stamp mismatches and only the missing package installs). - SqliteMemoryStore.select_fallback_embeddings()'s re-embed worklist now also matches rows with a non-empty embedding_model but no memories_vec row (a vector computed but never persisted because sqlite-vec was absent at write time), so such rows self-heal once the extension becomes available. Previously only the embedding_model='fallback' label was matched, and a correctly-labelled 'neural' row with no vector could never resurface. - doctor/check_setup gained an optional _sqlite_vector_search check that distinguishes "package not installed" from "installed but this Python's sqlite3 cannot load extensions" (different fixes: reinstall vs. a different Python build). - The embedding_upgrade consolidation cycle's result now also carries vectors_persisted: 0 and a reason when the store cannot persist vectors at all, so "upgraded: N" is never misread as "N vectors written". Decision recorded as ADR-1087 (wiki/adr/), per the decision-gate: large rationale blocks belong in the wiki, code keeps a short `# source: ADR-NNNN` pointer. New tests cover the worklist's vector-gap backfill and its exclusion of never-embedded rows, the embedding_upgrade counter's vectors_persisted flag, and the doctor check's optional status and package-vs-extension distinction. select_fallback_embeddings benchmarked at 3.1 ms/call on a 10k-row in-memory store with 200 vectorless rows seeded (100 returned per call, the existing limit). Full suite: 9119 passed, 11 skipped (darwin). Windows unverified in this environment; CI's ci-sqlite-min.txt job is the nearest proxy. Known: this branch's ADR-1087 and the sibling issue #633 fix branch's ADR-1087/1088 were assigned independently from the same tip and collide on number 1087 with different content. Whichever PR merges second must renumber (file, wiki/manifest.json entry, mirror under docs/adr/, and the `source: ADR-1087` pointers in this branch's scripts/launcher_pins.py and scripts/pip_constraint_sets.py). Co-Authored-By: Claude Sonnet 5 <[email protected]> * fix(sqlite): stamp 'neural' only after the vector actually lands Reviewed and reworked. The root cause the previous commit on this branch (715f580) never touched: SqliteMemoryStore._insert_memory_rows, reembed_memory, and update_memory_compression each stamp memories.embedding_model unconditionally whenever an embedding is present, whether or not the vec INSERT for it actually succeeds. _stamp_embedding_model's own docstring already stated the precondition every caller violated ("memory_id refers to a just-written vec row"). A memory written while sqlite-vec is unavailable -- the exact population issue #634 describes -- gets stamped 'neural' with no row in memories_vec behind it. The tag has two readings tangled into one write path. 'fallback' records which encoder ran; the re-embed worklist keys on it (WHERE embedding_model = 'fallback'), and a fallback memory with no vector must still surface for upgrade, so it stamps unconditionally -- issue #169's own tested contract (test_embedding_model_stamped_fallback) requires exactly this and is unchanged. 'neural' is read elsewhere (sqlite_store_search.py's mixed-store cross-rank filter) as a claim that a neural vector exists; stamping it with no vector behind it is issue #634's bug, in one word: 'neural' stamped without a vector. _insert_memory_rows and update_memory_compression now resolve the model name and whether the vec write actually succeeded, then stamp when model != "neural" or persisted: 'fallback' always stamps, 'neural' stamps only when memories_vec genuinely received the row. reembed_memory needed no such split -- its only caller, run_embedding_upgrade_cycle, now checks store.has_vec once before calling it at all, so a persisted vector is already guaranteed by the time it runs. run_embedding_upgrade_cycle gains has_vec on the _FallbackWorklistStore Protocol and returns {"upgraded": 0, "reason": ...} before encoding or restamping anything when has_vec is false, touching nothing -- the correct behavior for a cycle that cannot persist what it would compute, not a reason to restamp and report the gap after the fact. The vectors_persisted field the previous commit added to paper over that restamping is deleted; there is nothing left for it to report. select_fallback_embeddings's widened worklist query (the NOT IN memories_vec sweep) is kept: once the stamp fix lands, no production write path can produce a 'neural'-without-vector row going forward, so the sweep's remaining job is a one-time migration, self-healing installs whose database already carries rows the pre-fix bug wrote (the reporter's own measured 86 such rows). _sqlite_vector_search (mcp_server/doctor.py) changes from optional=True to optional=False: sqlite-vec now ships in every install's base package set, so its absence or a load failure is a broken install, not a supported degraded mode. Craftsmanship gate follow-up: this branch's first commit pushed mcp_server/doctor.py, embedding_upgrade.py, and tests_py/test_doctor.py over the method-size and file-size caps with inline rationale that belonged in the wiki ADR, not the code -- caught by this branch's own CI run once triggered. Docstrings trimmed to short pointers; the sqlite-vec-specific tests move to a new tests_py/test_doctor_sqlite_vector_search.py to keep test_doctor.py under 300 lines. tests_py/invariants/test_I2_canonical_writer.py's ALLOWED_WRITERS line numbers for sqlite_store.py shifted by +3 from this diff; updated to match (same three writers, confirmed by reading each site, not assumed). ADR renumbered 1087 -> 1089 (this branch's own number; sibling PRs #638 and #639 already claim 1088/1090) and rewritten around the actual defect ('neural' stamped without its vector) instead of only the install-list gap; the wiki ADR is now this fix's single source of truth for the 'fallback' vs 'neural' semantics, referenced by every call site that must not re-collapse them. Left unaddressed, flagged for the maintainer: _try_load_vec logs INFO and sets has_vec=False when the extension fails to load. Hardening it to raise at store-open time would change startup behavior on interpreters without loadable-extension support (common on macOS system/Framework Python) -- a product decision, not bundled here. Full suite: 9121 passed, 11 skipped (darwin, same pre-existing 11). ruff and the craftsmanship gate both clean. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik * fix(consolidation): declare has_vec as a property on _FallbackWorklistStore A bare has_vec: bool attribute annotation on the Protocol made pyright flag SqliteMemoryStore as unsafely overlapping the protocol (its own has_vec is a real @Property, not a data attribute), caught only once CI ran pull_request-triggered checks on this branch for the first time. Declaring has_vec as a Protocol @Property matches the real implementation and resolves the structural-overlap false positive. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik * fix(sqlite): downgrade to 'fallback' instead of skipping the stamp Review (PR #637) caught a real regression, worse than the bug this branch fixes: the gate `model != "neural" or persisted` skipped _stamp_embedding_model entirely when a neural encode had no persisted vector. The row's embedding_model stayed at the schema default '', which select_fallback_embeddings's own WHERE clause excludes -- the row became permanently invisible to the re-embed worklist, unlike the pre-fix bug (a plain 'neural' mislabel) that this same PR's vec-gap self-heal sweep already catches. Every insert/compression-update now always stamps something: a neural encode with no persisted vector downgrades to 'fallback' (recoverable, matches issue #169's unconditional-fallback contract); every other case stamps its own resolved model name. Updated the test that asserted the orphaned '' state as correct to assert the downgrade and that the row surfaces in the worklist instead. ADR-1089 and its docs mirror updated to describe the actual final behavior. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik * fix(tests): skip the neural-exclusion worklist test without sqlite-vec CI caught this on all four Test (Python 3.1x) matrix jobs, which don't install sqlite-vec (unlike the SQLite-backend job, which does): test_select_fallback_embeddings_worklist asserted a 'neural'-tagged insert is excluded from the fallback worklist, without has_vec being true. That assumption only ever held by accident, on the pre-fix code: a neural encode with no persisted vector used to be left unstamped (''), which is invisible to the worklist query for a different reason (the orphan bug this branch's own review just fixed). With the vector never actually able to persist, this test's "neural one" now correctly downgrades to 'fallback' and belongs in the worklist -- exactly test_neural_stamp_downgrades_to_fallback_without_a_persisted_vector's contract, already covered elsewhere in this file. Marked @_needs_vec, matching the file's existing convention for assertions that are only meaningful when sqlite-vec is actually available. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik --------- Co-authored-by: Claude Sonnet 5 <[email protected]>
Summary
Two Windows-reported defects, both cross-platform in the underlying code paths even though only observed there. First,
install-plugin.shwrote~/.claude/methodology/backend.jsononly after runningscripts/setup.py, guarded by|| fail; any setup failure discarded the already-correct Phase 0 backend decision, and the next launch's_detect_runtime()/_construct_store()silently forcedbackend="postgresql"for a chosen SQLite install with none ever provisioned. Second,scripts/setup.py'scache_embedding_model()and its shell twinscripts/lib/precache_embedding_model.shspawn a child process that cannot isolatedeps/from user site-packages the waylauncher_site.isolate_deps()does in-process (issue #621), so a conflicting user-site torch/torchaudio build could still abort the embedding-model pre-cache.Closes #633
Type of change
Test plan
ci-sqlite-min.txtjob is the nearest proxy.test_install_plugin_marker_order.pydrives the real installer against apython3stub that fails onlyscripts/setup.py, proving the marker survives (verified to fail on the pre-fix script and pass on the fixed one);_model_cache_child_source()compiled and run end-to-end against a stubbedsentence_transformers;repr()path-escaping verified against Windows-shaped paths (backslashes, an apostrophe).Audit notes
setup_database()was audited (not assumed) to confirm it needs noisolate_deps()bootstrap, since its child only importspsycopg/pg_schema, not the ML stack; documented in the commit._model_checks()widenedImportErrortoExceptionto match_postgres_checks()'s existing pattern, closing an "exited before reporting any check" failure mode on a torch ABI mismatch (OSError/RuntimeError).Coding-standards compliance
any/ untyped dicts at boundaries# source: ADR-1087/ADR-1088pointersBreaking changes
None.
Reviewer checklist
Known issue: ADR number collision
This branch's
ADR-1087/ADR-1088(wiki/adr/) and the siblingfix/issue-634-sqlite-vecbranch'sADR-1087were assigned independently from the same tip and collide on number 1087 with different content. Whichever PR merges second must renumber: the wiki ADR file(s), thewiki/manifest.jsonentry, thedocs/adr/mirror, and the# source: ADR-1087pointers in this branch'sscripts/setup.pyandscripts/lib/precache_embedding_model.sh.🤖 Generated with Claude Code