Skip to content

fix(setup): persist backend marker before setup.py can fail; close ML-child isolation gap - #638

Merged
cdeust merged 5 commits into
mainfrom
fix/issue-633-backend-marker
Sep 23, 2026
Merged

cdeust merged 5 commits into
mainfrom
fix/issue-633-backend-marker

Conversation

@cdeust

@cdeust cdeust commented Sep 22, 2026

Copy link
Copy Markdown
Owner

Summary

Two Windows-reported defects, both cross-platform in the underlying code paths even though only observed there. First, install-plugin.sh wrote ~/.claude/methodology/backend.json only after running scripts/setup.py, guarded by || fail; any setup failure discarded the already-correct Phase 0 backend decision, and the next launch's _detect_runtime()/_construct_store() silently forced backend="postgresql" for a chosen SQLite install with none ever provisioned. Second, scripts/setup.py's cache_embedding_model() and its shell twin scripts/lib/precache_embedding_model.sh spawn a child process that cannot isolate deps/ from user site-packages the way launcher_site.isolate_deps() does in-process (issue #621), so a conflicting user-site torch/torchaudio build could still abort the embedding-model pre-cache.

Closes #633

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Test plan

  • All existing tests pass: full suite 9117 passed, 11 skipped (darwin). Windows unverified in this environment; CI's ci-sqlite-min.txt job is the nearest proxy.
  • New tests added for new behavior: test_install_plugin_marker_order.py drives the real installer against a python3 stub that fails only scripts/setup.py, proving the marker survives (verified to fail on the pre-fix script and pass on the fixed one); _model_cache_child_source() compiled and run end-to-end against a stubbed sentence_transformers; repr() path-escaping verified against Windows-shaped paths (backslashes, an apostrophe).
  • Mutation survival check: the marker-order regression test fails against the unpatched script, so a mutant reordering the write-then-run sequence back would be caught.
  • Manual verification of any UI / CLI / MCP-tool behavior changes: not exercised manually on Windows.

Audit notes

  • Engineering review: setup_database() was audited (not assumed) to confirm it needs no isolate_deps() bootstrap, since its child only imports psycopg/pg_schema, not the ML stack; documented in the commit.
  • _model_checks() widened ImportError to Exception to match _postgres_checks()'s existing pattern, closing an "exited before reporting any check" failure mode on a torch ABI mismatch (OSError/RuntimeError).
  • Outstanding deferred findings: none.

Coding-standards compliance

  • §2.2 Layer dependency direction preserved
  • §3.2 No any / untyped dicts at boundaries
  • §4.1 No file > 500 lines
  • §4.2 No function > 50 lines
  • §4.4 No function with > 4 parameters
  • §7 Local reasoning preserved
  • §8 Constants carry # source: ADR-1087/ADR-1088 pointers
  • §9 No dead code, no unreferenced TODOs

Breaking changes

None.

Reviewer checklist

  • CHANGELOG.md updated under the appropriate section.
  • Documentation updated (README / SKILL.md / docs/).
  • No secrets / credentials / PII in the diff.
  • CI passes on the latest commit.

Known issue: ADR number collision

This branch's ADR-1087/ADR-1088 (wiki/adr/) and the sibling fix/issue-634-sqlite-vec branch's ADR-1087 were assigned independently from the same tip and collide on number 1087 with different content. Whichever PR merges second must renumber: the wiki ADR file(s), the wiki/manifest.json entry, the docs/adr/ mirror, and the # source: ADR-1087 pointers in this branch's scripts/setup.py and scripts/lib/precache_embedding_model.sh.

🤖 Generated with Claude Code

…-child isolation gap

Two Windows-reported defects (issue #633), both cross-platform in the
underlying pin lists / code paths even though only observed there:

1. install-plugin.sh wrote ~/.claude/methodology/backend.json only after
   running scripts/setup.py, guarded by `|| fail`. Any setup failure
   discarded the already-correct Phase 0 backend decision. On the next
   launch, memory_config._detect_runtime() defaults to "cli" (the plugin's
   own CORTEX_RUNTIME="" resolves there), and memory_store._construct_store()
   forces backend="postgresql" whenever runtime=="cli" and backend=="auto",
   so a missing marker silently converted a chosen SQLite install into a
   hard PostgreSQL requirement, with none ever provisioned. Fixed: the
   marker now persists immediately after the backend is decided, before
   setup can fail and exit.

2. scripts/setup.py's cache_embedding_model() and its macOS/Linux twin
   scripts/lib/precache_embedding_model.sh spawn a child process that
   reaches deps/ only through PYTHONPATH, which cannot cut user
   site-packages the way launcher_site.isolate_deps() does in-process
   (issue #621), so a conflicting user-site torch/torchaudio build could
   still abort the embedding-model pre-cache. Both now bootstrap
   isolate_deps() in the child before importing sentence_transformers.
   scripts/setup.py's cache_embedding_model payload is pulled into a
   standalone, unit-tested _model_cache_child_source() function.
   _model_checks() also widens ImportError to Exception, matching
   _postgres_checks()'s own pattern, so a torch ABI mismatch (OSError or
   RuntimeError) reports a [FAIL] row instead of crashing verify() outright,
   closing the "exited before reporting any check" failure mode.

setup_database() needed no such bootstrap (its child imports only psycopg
and pg_schema, not the ML stack); audited and documented rather than
assumed.

Decisions recorded as ADR-1087 and ADR-1088 (wiki/adr/), per the
decision-gate: large rationale blocks belong in the wiki, code keeps a
short `# source: ADR-NNNN` pointer.

New regression test (test_install_plugin_marker_order.py) drives the real
installer against a python3 stub that fails only scripts/setup.py,
proving the marker survives, verified to fail on the pre-fix script and
pass on the fixed one. New tests compile and run
_model_cache_child_source() end-to-end against a stubbed
sentence_transformers, and prove repr() escapes Windows-shaped paths
(backslashes, an apostrophe) correctly.

Full suite: 9117 passed, 11 skipped (darwin). Windows unverified in this
environment; CI's ci-sqlite-min.txt job is the nearest proxy.

Known: this branch's ADR-1087/1088 and the sibling issue #634 fix branch's
ADR-1087 were assigned independently from the same tip and collide on
number 1087 with different content. Whichever PR merges second must
renumber (file, wiki/manifest.json entry, mirror under docs/adr/, and the
`source: ADR-1087` pointers in this branch's scripts/setup.py and
scripts/lib/precache_embedding_model.sh).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
cdeust added a commit that referenced this pull request Sep 23, 2026
Reviewed and reworked. The root cause the previous commit on this branch
(715f580) never touched: SqliteMemoryStore._insert_memory_rows,
reembed_memory, and update_memory_compression each stamp
memories.embedding_model unconditionally whenever an embedding is
present, whether or not the vec INSERT for it actually succeeds.
_stamp_embedding_model's own docstring already stated the precondition
every caller violated ("memory_id refers to a just-written vec row").
A memory written while sqlite-vec is unavailable -- the exact population
issue #634 describes -- gets stamped 'neural' with no row in
memories_vec behind it.

The tag has two readings tangled into one write path. 'fallback' records
which encoder ran; the re-embed worklist keys on it
(WHERE embedding_model = 'fallback'), and a fallback memory with no
vector must still surface for upgrade, so it stamps unconditionally --
issue #169's own tested contract (test_embedding_model_stamped_fallback)
requires exactly this and is unchanged. 'neural' is read elsewhere
(sqlite_store_search.py's mixed-store cross-rank filter) as a claim that
a neural vector exists; stamping it with no vector behind it is issue
#634's bug, in one word: 'neural' stamped without a vector.

_insert_memory_rows and update_memory_compression now resolve the model
name and whether the vec write actually succeeded, then stamp when
model != "neural" or persisted: 'fallback' always stamps, 'neural'
stamps only when memories_vec genuinely received the row.
reembed_memory needed no such split -- its only caller,
run_embedding_upgrade_cycle, now checks store.has_vec once before
calling it at all, so a persisted vector is already guaranteed by the
time it runs.

run_embedding_upgrade_cycle gains has_vec on the _FallbackWorklistStore
Protocol and returns {"upgraded": 0, "reason": ...} before encoding or
restamping anything when has_vec is false, touching nothing -- the
correct behavior for a cycle that cannot persist what it would compute,
not a reason to restamp and report the gap after the fact. The
vectors_persisted field the previous commit added to paper over that
restamping is deleted; there is nothing left for it to report.

select_fallback_embeddings's widened worklist query (the NOT IN
memories_vec sweep) is kept: once the stamp fix lands, no production
write path can produce a 'neural'-without-vector row going forward, so
the sweep's remaining job is a one-time migration, self-healing installs
whose database already carries rows the pre-fix bug wrote (the
reporter's own measured 86 such rows).

_sqlite_vector_search (mcp_server/doctor.py) changes from optional=True
to optional=False: sqlite-vec now ships in every install's base package
set, so its absence or a load failure is a broken install, not a
supported degraded mode.

Craftsmanship gate follow-up: this branch's first commit pushed
mcp_server/doctor.py, embedding_upgrade.py, and tests_py/test_doctor.py
over the method-size and file-size caps with inline rationale that
belonged in the wiki ADR, not the code -- caught by this branch's own
CI run once triggered. Docstrings trimmed to short pointers; the
sqlite-vec-specific tests move to a new tests_py/test_doctor_sqlite_vector_search.py
to keep test_doctor.py under 300 lines.

tests_py/invariants/test_I2_canonical_writer.py's ALLOWED_WRITERS line
numbers for sqlite_store.py shifted by +3 from this diff; updated to
match (same three writers, confirmed by reading each site, not assumed).

ADR renumbered 1087 -> 1089 (this branch's own number; sibling PRs #638
and #639 already claim 1088/1090) and rewritten around the actual
defect ('neural' stamped without its vector) instead of only the
install-list gap; the wiki ADR is now this fix's single source of truth
for the 'fallback' vs 'neural' semantics, referenced by every call site
that must not re-collapse them.

Left unaddressed, flagged for the maintainer: _try_load_vec logs INFO
and sets has_vec=False when the extension fails to load. Hardening it
to raise at store-open time would change startup behavior on
interpreters without loadable-extension support (common on macOS
system/Framework Python) -- a product decision, not bundled here.

Full suite: 9121 passed, 11 skipped (darwin, same pre-existing 11).
ruff and the craftsmanship gate both clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
@cdeust cdeust closed this Sep 23, 2026
@cdeust cdeust reopened this Sep 23, 2026
@cdeust cdeust closed this Sep 23, 2026
@cdeust
cdeust deleted the fix/issue-633-backend-marker branch September 23, 2026 09:01
@cdeust
cdeust restored the fix/issue-633-backend-marker branch September 23, 2026 09:02
@cdeust cdeust reopened this Sep 23, 2026
cdeust and others added 3 commits September 23, 2026 11:24
…late

Caught by this branch's own craftsmanship gate once a real CI run finally
ran on it: test_marker_persists_backend_on_a_successful_sqlite_install
exceeded the 40-line method cap, duplicating the stub-writing and
env-building steps already in the sibling failure-path test almost
verbatim.

Extracted _write_setup_py_stub(bin_dir, exit_code=...) (the prior
failure-only stub, now parametrized on the stubbed exit code) and
_run_installer(tmp_path, setup_py_exit_code=...) (the shared
home/PATH-isolated subprocess.run call), plus a small _marker_backend()
reader both tests' final assertions were duplicating too. Both tests now
read as their actual distinguishing behavior (exit code in, marker
survives or not) instead of repeating the installer-driving mechanics.

No behavior change: both tests still pass unchanged.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
install-plugin.sh's stale-install pruning (phase 2b) runs pip3 show/
uninstall unconditionally against whatever pip3 resolves to on PATH.
This test only stubbed python3, so in CI pip3 resolved to the real
interpreter's pip and genuinely uninstalled this repo's own
just-installed editable hypermnesia-mcp package, breaking the
"Verify hook-free MCP host contract" step later in the same job
(FileNotFoundError: hypermnesia-mcp). Reproduced deterministically via
a CI rerun of PR #638 (same two jobs failed identically both times).

Stub pip/pip3 as always-report-not-installed, matching the existing
python3 stub pattern, so the pruning phase's uninstall branch is never
reached.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
@cdeust

cdeust commented Sep 23, 2026

Copy link
Copy Markdown
Owner Author

ZETETIC-REVIEW: REQUEST_CHANGES

Stakes: High (install/launch-path correctness gate for every plugin user; touches the sole determinant of which storage backend a launch resolves to).

What holds up. The primary fix is correctly implemented: I traced both branches in scripts/install-plugin.sh (sqlite via run_setup_py, postgres via setup.py-on-Windows or setup.sh-on-Darwin/Linux) and the marker write now sits before every fail-and-exit path in all three. The except Exception widening in scripts/setup.py::_model_checks is scoped tightly, one import statement per try block, matching the existing _postgres_checks pattern exactly — it does not swallow anything outside the two named optional-dependency probes. The pip/pip3 stub fix in tests_py/scripts/test_install_plugin_marker_order.py is sound: it makes pip3 show <pkg> fail so phase 2b's uninstall branch is never reached, and it does not touch the test's actual subject (marker-write ordering around setup.py's exit code). CI is green across all jobs including the new Windows SQLite job.

Two problems in the diff itself.

  1. The reordering in install-plugin.sh regresses backend-switch-then-fail. Trace it: an existing working sqlite install has marker="sqlite". User re-runs with --postgres (or CORTEX_BACKEND=postgres) to switch. Phase 0 sets BACKEND=postgresql, the marker is now overwritten to postgresql before setup.sh runs. setup.sh fails (no local PostgreSQL yet) and the script exits under set -e. The marker now reads postgresql while the only working store is sqlite — the next launch forces PostgreSQL with none provisioned, reproducing the exact database_not_connected symptom this PR is fixing, just in the reverse direction. Before this PR, the write-after-setup ordering happened to protect this case by accident (a failed setup left the old, working marker alone). ADR-1088's "the marker records a decision, not a completed install, and a marker that says sqlite but whose setup failed is still strictly better than no marker" is true for the no-prior-marker case (Upgrade 3.x → 4.x leaves the plugin on PostgreSQL: backend.json is never written when setup.py fails #633's actual repro) but false when overwriting an existing, working marker with a different backend — it directly contradicts the script's own documented invariant ("an upgrade never silently downgrades a Postgres install to SQLite" / "Idempotent. Safe to re-run."). Minimum fix: only pre-write the marker when there was no prior marker, or when the chosen backend matches the prior one; otherwise defer the write until setup succeeds, same as before. Add a test: pre-seed a postgresql marker, set CORTEX_BACKEND=sqlite, fail setup.py, assert the marker still reads postgresql.

  2. scripts/lib/precache_embedding_model.sh interpolates ${project_dir} and ${deps_dir} directly into single-quoted Python string literals (sys.path.insert(0, '${project_dir}/scripts'), launcher_site.isolate_deps('${deps_dir}')) instead of using repr()-style escaping. ADR-1087 explicitly states the decision as "Paths are embedded via repr(), not an f-string, so a Windows path's backslashes escape correctly inside the child's source" and even ships a dedicated test for that exact O'Brien-apostrophe shape — but only for the Python side (scripts/setup.py::_model_cache_child_source). The bash twin does not apply the same escaping and will emit a Python SyntaxError for any project/deps path containing a single quote. It fails soft (the step already warns-and-continues on subprocess failure), so it degrades rather than crashes the install, but it contradicts the ADR's own stated decision and leaves the fix half-applied. Fix: pass the paths through environment variables and read them with os.environ in the child, matching the pattern already used for the backend marker block in this same PR.

Non-blocking: _model_checks's new except Exception arms discard the actual exception (OSError with the libtorchaudio message, etc.) — before this change an uncaught exception at least surfaced a traceback; now the user gets a bare [FAIL] row with no cause. Consider warn(f"...: {exc!r}") in each arm. Not blocking since it matches the existing _postgres_checks pattern.

Everything else — layering, SRP/OCP on the extracted _model_cache_child_source, wiring, ADR pointers, craftsmanship-gate size limits — is clean.

Review (PR #638) caught a real regression: persisting the marker
before setup runs (this PR's own #633 fix) unconditionally overwrote
a previously-persisted, working backend choice with the newly
requested one before that new backend was ever confirmed working. A
user switching from a working sqlite install to --postgres, whose
postgres setup then fails, ended up with the marker pointing at
postgresql with nothing provisioned -- issue #633's exact bug class,
in reverse.

Capture the marker's previous backend before overwriting it, and
restore it whenever the newly requested backend differs and its setup
fails. A fresh install (no prior marker) or a same-backend reinstall
have nothing to restore, so #633's original scenario is untouched.
Added a regression test that seeds a working sqlite marker, requests
a postgres switch through the Windows branch (uname stubbed, which
reuses the same stubbable scripts/setup.py path already used for the
sqlite case), fails setup, and asserts the marker still reads sqlite.

Also fixed: scripts/lib/precache_embedding_model.sh interpolated
project_dir/deps_dir directly into single-quoted Python string
literals, unlike its Python twin (_model_cache_child_source), which
escapes via repr() per ADR-1087. A path containing a single quote or
backslash would break the embedded Python source. Pass both paths
through env vars instead, which need no escaping at all.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik
@cdeust

cdeust commented Sep 23, 2026

Copy link
Copy Markdown
Owner Author

ZETETIC-REVIEW: APPROVE

Both blocking findings from the previous review are resolved in c43eecb.

Finding 1 (marker-write regression on backend switch), resolved. Verified by mutation: I checked out the pre-fix commit (fd7499e) and ran the new test test_marker_restores_previous_backend_when_switch_setup_fails against it, and it fails (assert 'postgresql' == 'sqlite'), confirming the marker really was left pointing at an unconfirmed backend before this fix. Against the current head, all 20 tests in tests_py/scripts/test_install_plugin_marker_order.py and test_setup_py_backend_skip.py pass. restore_previous_marker() is called before fail() on all three failure branches (sqlite, Windows-postgres, unsupported-OS), and the Darwin/Linux bash scripts/setup.sh call is covered too: scripts/setup.sh has set -euo pipefail at the top, so a real failure there does propagate a non-zero exit for the new || { restore_previous_marker; fail ...; } to catch (this branch itself isn't exercised by the new test, since it needs a real PostgreSQL, but the exit-code contract that makes it correct is confirmed).

Finding 2 (bash string-interpolation escaping), resolved. Verified by mutation: I ran both the old and new scripts/lib/precache_embedding_model.sh against a project_dir containing an apostrophe (/tmp/o'brien test). The old version raises a SyntaxError inside the embedded Python (sys.path.insert(0, '/tmp/o'brien test/scripts')); the new version, which passes the path only through CORTEX_PRECACHE_SCRIPTS_DIR/CORTEX_PRECACHE_DEPS_DIR env vars read via os.environ[...], succeeds cleanly. Existing test_precache_embedding_model_step.py still passes.

CI is green across the board: Lint, Craftsmanship Gate, Type Check, Test 3.10 through 3.13, Windows SQLite, Docker Smoke, Fuzz.

Non-blocking follow-ups (do not block merge):

  1. ADR-1088's Decision section still says the marker write does not depend on setup succeeding. The new restore-on-failed-switch behavior makes that true only for same-backend or fresh-install cases, not for a backend switch. Worth a short amendment to the ADR (and its wiki mirror) so the decision index stays accurate, per the wiki-is-the-decision-index convention.
  2. Fix 2 (bash escaping) has no automated regression test; only the Python twin (_model_cache_child_source) got one. Consider adding a case to test_precache_embedding_model_step.py that passes a project_dir containing an apostrophe, mirroring the manual check above.
  3. restore_previous_marker()'s 2>/dev/null || true swallows a write failure and then unconditionally prints "restored working marker to $PREVIOUS_BACKEND". If the restore write itself fails, that message would be false. Low probability, but worth gating the warn on the write's actual success rather than || true-ing past it.

@cdeust
cdeust merged commit 85200ff into main Sep 23, 2026
28 checks passed
@cdeust
cdeust deleted the fix/issue-633-backend-marker branch September 23, 2026 11:30
cdeust added a commit that referenced this pull request Sep 23, 2026
…ll (#639)

* fix(consolidate): turn silent PG-only crashes into named skips on SQLite

Windows-reported (issue #636), but the root cause is not platform-specific:
consolidate's wiki maintenance cycle runs four passes on every invocation
(wiki_source_backfill_pass, wiki_domain_backfill_pass,
wiki_citation_seed_pass, wiki_backlog_pass's lesson-promotion count), each
needing store.batch_pool (a psycopg_pool.ConnectionPool only PgMemoryStore
provides, since the SQL is Postgres-schema-qualified: wiki.pages,
wiki.citations, %s placeholders). SqliteMemoryStore has no batch_pool
attribute at all, so every one of these passes raised AttributeError on
every SQLite install, on every run.

Three of the four caught that AttributeError internally and returned
{"status": "error: ..."} instead of letting it propagate.
wiki_maintenance.py's own escalation only fired on a RAISE past the
await, never on a returned error stanza, so consolidate kept reporting
status: "ok" and failed_stages: [] while four maintenance passes were
silently broken on the default backend, on every run. The fourth pass
already degraded to None on the same AttributeError but mischaracterized
SQLite as "fallback in tests", and called silent_failure.note() on every
occurrence, incrementing cortex_silent_failures_total on every
consolidate run for the lifetime of a SQLite install for a gap that
isn't actually a failure.

Fixed:
- Added mcp_server/handlers/consolidation/batch_pool_capability.py, a
  shared batch_pool_skip_reason(store) guard. Every batch_pool-gated pass
  now checks it before entering its try block and returns
  {"status": "skipped: store has no batch_pool (non-PostgreSQL
  backend)"} instead of ever reaching AttributeError. Applied to the
  three consolidate-wired wiki passes and, since audit confirmed the
  identical unconditional store.batch_pool pattern, to the five
  standalone-campaign passes the issue flagged as unverified suspects
  (memory_dedup_exact_pass, memory_domain_backfill_pass,
  memory_reheat_pass, near_dup_calibration_pass's two entry points,
  write_class_backfill_pass).
- wiki_maintenance.py gained _escalate_if_error(): it now also escalates
  out["status"] when a sub-pass RETURNS (not just raises) an "error:"
  status, fixing the previously dead escalation path. A "skipped: ..."
  status does not escalate.
- wiki_backlog_pass._lesson_promotion_backlog checks the same guard
  first and returns None without calling silent_failure.note() when the
  capability is simply absent, reserving that call for a genuine query
  failure once batch_pool exists. Docstring no longer calls SQLite a
  test-only fallback.

Decision recorded as ADR-1089 (wiki/adr/), per the decision-gate. Not
1087: two sibling branches from this same reporting session
(fix/issue-634-sqlite-vec, fix/issue-633-backend-marker; PRs #637, #638)
already claimed 1087 and 1088 independently from the same main tip, and
both document that collision against each other. 1089 avoids adding a
third branch to it; whichever PR merges last still renumbers as needed.

Left out of scope, flagged for a separate decision: ingest_codebase.py's
entity/edge ingestion also references store.batch_pool unconditionally,
with no try/except at all, but its sink SQL uses PostgreSQL's COPY ...
FROM STDIN wire protocol (mcp_server/infrastructure/staging_resolve_sink.py),
which has no SQLite equivalent. This is a materially larger, pre-existing,
PG-only architectural gap (crashes loudly today, not silently), not a
simple missing-guard bug.

New tests: unit coverage for the guard helper; a contract test running
all nine fixed passes against a real, fresh SqliteMemoryStore built
through the composition root, asserting every one degrades to
"skipped: ..." and never "error: ..."; wiki_maintenance tests proving a
returned (non-raised) error stanza now escalates, and that a skipped
stanza does not.

Full suite: 9157 passed, 11 skipped (darwin). Windows unverified in this
environment.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* fix(consolidate): gate PG-only wiki passes once at wiring, not per call

Reworks the previous commit (efac006) on this branch. That commit fixed
issue #636's reported symptom (consolidate reports status ok while four
wiki passes silently fail on SQLite) by adding a hasattr(store,
"batch_pool") guard inside each of nine call sites, turning the
AttributeError into a returned {"status": "skipped: ..."} stanza.
Reviewed and rejected: that is a catch-and-degrade shim restating the
same defect with a softer word, not a fix at the root, and the repo's
own non-negotiable already says so ("no silent fallbacks or
compatibility shims").

This commit fixes the actual root cause instead: wiki_maintenance.py
wired four PostgreSQL-only passes (wiki_source_backfill_pass,
wiki_domain_backfill_pass, wiki_citation_seed_pass,
wiki_backlog_pass's lesson-promotion count) unconditionally, regardless
of backend. run_wiki_maintenance now decides once, at the top, whether
store is PostgreSQL-backed (hasattr(store, "batch_pool"), the same
idiom this file already used for iter_memories_for_decay) and only
invokes those four inside that branch. On SQLite they are simply absent
from the response, the same way "wiki" itself is absent from
consolidate's stats when wiki=False -- not reported as failed, not
reported as skipped.

The four passes no longer catch their own AttributeError (or any
exception) for a missing-capability case: run_wiki_maintenance is each
one's sole caller now, so being invoked without batch_pool is a wiring
bug and raises, like any other broken precondition. This also revives
wiki_maintenance's existing except blocks around each await as their
real, only error boundary -- previously dead code, since every pass
swallowed its own exception before the caller ever saw it.

wiki_backlog_pass._lesson_promotion_backlog moves out of
run_backlog_pass (unconditional, filesystem-based) into
run_wiki_maintenance's own PostgreSQL branch; its try/except and
silent_failure.note for a genuine query failure are unchanged
(ADR-0640's accepted pattern), only the call site moved, which is what
stops the metric from incrementing on every SQLite consolidate run.

The five standalone campaign scripts sharing the identical
store.batch_pool pattern (memory_dedup_exact_pass,
memory_domain_backfill_pass, memory_reheat_pass,
near_dup_calibration_pass's two entry points,
write_class_backfill_pass) are not wired into consolidate and keep
their original, pre-#636 try/except (a human runs the matching
scripts/*.py CLI directly and reads the printed status) unchanged. Each
script's entry point now checks hasattr(store, "batch_pool") once,
before calling its pass, and exits 1 with a one-line message when
absent.

Deleted batch_pool_capability.py and its unit test (the shim and its
own test); deleted test_wiki_batch_pool_sqlite_contract.py (tested the
shim's "skipped:" status shape, no longer applicable). New tests in
test_wiki_maintenance.py: the four PG-only keys are absent on a store
without batch_pool and present on one with it; a contract test runs the
whole cycle against a real, fresh SqliteMemoryStore built through the
composition root and asserts no stanza anywhere has an "error:" status
and the four keys are absent -- issue #636's own requested check.

consolidate's tool schema documents the four keys as PostgreSQL-only.

ADR renumbered 1089 -> 1090 (this branch's own number; the sibling PRs
#637/#638 already claim 1087/1088) and rewritten to describe this
design, including why the first (shim) version was rejected on review.

Left out of scope, flagged separately: ingest_codebase.py's entity/edge
ingestion also references store.batch_pool unconditionally, but its
sink SQL uses PostgreSQL's COPY ... FROM STDIN wire protocol with no
SQLite equivalent -- a materially larger, pre-existing, architectural
gap, not a missing-guard bug.

Full suite: 9146 passed, 11 skipped (darwin, same pre-existing 11 as
before this branch). ruff check/format and the craftsmanship gate both
clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

* fix(consolidate): escalate lesson_promotion_backlog failures like the others

Advisor review of the previous commit (f9398f1) on this branch found
one remaining inconsistency: _lesson_promotion_backlog kept its own
internal try/except, so a genuine PostgreSQL query failure (pool
exhausted, table missing) degraded to lesson_promotion_backlog: None
while out["status"] stayed "ok" -- the exact shape issue #636 reported,
just for one key instead of four, and the one PG-only pass this branch's
rework had not yet made consistent with the other three.

Removed the try/except from _lesson_promotion_backlog: it now raises on
a query failure like source_backfill/domain_backfill/citation_seed do.
run_wiki_maintenance wraps the call in its own try/except and escalates
out["status"] to "lesson_promotion_backlog_error: ..." on failure,
matching the other three exactly. This also drops the silent_failure.note
call the function no longer needs -- the wiring fix from the previous
commit already means it is never invoked at all on a non-PostgreSQL
store, which is what actually stops the metric pollution, not a
swallowed exception.

Regenerated .craftsmanship-baseline.json: removing wiki_backlog_pass.py's
now-unused `mcp_server.observability` import fixed a pre-existing
layer-violation baseline entry, which needed pruning as stale.

New tests: wiki_backlog_pass's own unit tests now assert raise instead
of a degraded None; a new wiki_maintenance test proves a real failure
there escalates status exactly like the citation-seed equivalent.

Full suite: 9147 passed, 11 skipped (darwin, same pre-existing 11). ruff
and the craftsmanship gate both clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

* fix(tests): build the contract test's store through the real composition root

Review (PR #639) caught the contract test (TestRealSqliteStoreContract,
issue #636's own requested check) instantiating SqliteMemoryStore(path)
directly, bypassing get_shared_store/MemoryStore/_construct_store --
exactly the backend-selection logic this bug is about. Rebuilt it to
force CORTEX_MEMORY_STORE_BACKEND=sqlite and go through
get_shared_store(), clearing the memory-settings cache and shared-store
singleton before and after, matching the pattern already used in
tests_py/hooks/test_entry_backend_parity.py.

Also fixed the non-blocking duplication finding: the five campaign
scripts (memory_dedup_exact, memory_domain_backfill, memory_reheat,
near_dup_calibrate, backfill_write_class) each carried an identical
7-line batch_pool guard. Extracted to
mcp_server/shared/campaign_store_guard.py::require_postgres_store,
behavior-preserving.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>
cdeust added a commit that referenced this pull request Sep 23, 2026
…s rows (#637)

* fix(sqlite): install sqlite-vec on every backend, self-heal vectorless rows

Windows-reported (issue #634), but the root cause is not platform-specific:
neither dependency-install path the plugin uses ever named the [sqlite]
extra.

- scripts/launcher_pins.py::BASE_PACKAGES (installed by ensure_deps() on
  every hook/session-start launch) listed the PostgreSQL drivers
  unconditionally but nothing from [sqlite].
- requirements/setup.txt, generated for extras (postgresql, codebase,
  benchmarks), is what scripts/setup.py::install_deps() installs on BOTH
  backends (SKIP_POSTGRES does not change which file is read).

Consequence: the SQLite backend, the zero-config default, ran with vector
search silently disabled (FTS-only) on every install, on every OS, while
check_setup/doctor reported the store fully ready.

Fixed:
- Added "sqlite" to setup.txt's generated extras and sqlite-vec==0.1.9 to
  BASE_PACKAGES, pinned to match. Existing installs self-heal on next
  launch (ensure_deps()'s pin-satisfaction check compares sorted(pins), so
  the stamp mismatches and only the missing package installs).
- SqliteMemoryStore.select_fallback_embeddings()'s re-embed worklist now
  also matches rows with a non-empty embedding_model but no memories_vec
  row (a vector computed but never persisted because sqlite-vec was absent
  at write time), so such rows self-heal once the extension becomes
  available. Previously only the embedding_model='fallback' label was
  matched, and a correctly-labelled 'neural' row with no vector could
  never resurface.
- doctor/check_setup gained an optional _sqlite_vector_search check that
  distinguishes "package not installed" from "installed but this Python's
  sqlite3 cannot load extensions" (different fixes: reinstall vs. a
  different Python build).
- The embedding_upgrade consolidation cycle's result now also carries
  vectors_persisted: 0 and a reason when the store cannot persist vectors
  at all, so "upgraded: N" is never misread as "N vectors written".

Decision recorded as ADR-1087 (wiki/adr/), per the decision-gate: large
rationale blocks belong in the wiki, code keeps a short
`# source: ADR-NNNN` pointer.

New tests cover the worklist's vector-gap backfill and its exclusion of
never-embedded rows, the embedding_upgrade counter's vectors_persisted
flag, and the doctor check's optional status and package-vs-extension
distinction. select_fallback_embeddings benchmarked at 3.1 ms/call on a
10k-row in-memory store with 200 vectorless rows seeded (100 returned per
call, the existing limit).

Full suite: 9119 passed, 11 skipped (darwin). Windows unverified in this
environment; CI's ci-sqlite-min.txt job is the nearest proxy.

Known: this branch's ADR-1087 and the sibling issue #633 fix branch's
ADR-1087/1088 were assigned independently from the same tip and collide
on number 1087 with different content. Whichever PR merges second must
renumber (file, wiki/manifest.json entry, mirror under docs/adr/, and the
`source: ADR-1087` pointers in this branch's scripts/launcher_pins.py and
scripts/pip_constraint_sets.py).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* fix(sqlite): stamp 'neural' only after the vector actually lands

Reviewed and reworked. The root cause the previous commit on this branch
(715f580) never touched: SqliteMemoryStore._insert_memory_rows,
reembed_memory, and update_memory_compression each stamp
memories.embedding_model unconditionally whenever an embedding is
present, whether or not the vec INSERT for it actually succeeds.
_stamp_embedding_model's own docstring already stated the precondition
every caller violated ("memory_id refers to a just-written vec row").
A memory written while sqlite-vec is unavailable -- the exact population
issue #634 describes -- gets stamped 'neural' with no row in
memories_vec behind it.

The tag has two readings tangled into one write path. 'fallback' records
which encoder ran; the re-embed worklist keys on it
(WHERE embedding_model = 'fallback'), and a fallback memory with no
vector must still surface for upgrade, so it stamps unconditionally --
issue #169's own tested contract (test_embedding_model_stamped_fallback)
requires exactly this and is unchanged. 'neural' is read elsewhere
(sqlite_store_search.py's mixed-store cross-rank filter) as a claim that
a neural vector exists; stamping it with no vector behind it is issue
#634's bug, in one word: 'neural' stamped without a vector.

_insert_memory_rows and update_memory_compression now resolve the model
name and whether the vec write actually succeeded, then stamp when
model != "neural" or persisted: 'fallback' always stamps, 'neural'
stamps only when memories_vec genuinely received the row.
reembed_memory needed no such split -- its only caller,
run_embedding_upgrade_cycle, now checks store.has_vec once before
calling it at all, so a persisted vector is already guaranteed by the
time it runs.

run_embedding_upgrade_cycle gains has_vec on the _FallbackWorklistStore
Protocol and returns {"upgraded": 0, "reason": ...} before encoding or
restamping anything when has_vec is false, touching nothing -- the
correct behavior for a cycle that cannot persist what it would compute,
not a reason to restamp and report the gap after the fact. The
vectors_persisted field the previous commit added to paper over that
restamping is deleted; there is nothing left for it to report.

select_fallback_embeddings's widened worklist query (the NOT IN
memories_vec sweep) is kept: once the stamp fix lands, no production
write path can produce a 'neural'-without-vector row going forward, so
the sweep's remaining job is a one-time migration, self-healing installs
whose database already carries rows the pre-fix bug wrote (the
reporter's own measured 86 such rows).

_sqlite_vector_search (mcp_server/doctor.py) changes from optional=True
to optional=False: sqlite-vec now ships in every install's base package
set, so its absence or a load failure is a broken install, not a
supported degraded mode.

Craftsmanship gate follow-up: this branch's first commit pushed
mcp_server/doctor.py, embedding_upgrade.py, and tests_py/test_doctor.py
over the method-size and file-size caps with inline rationale that
belonged in the wiki ADR, not the code -- caught by this branch's own
CI run once triggered. Docstrings trimmed to short pointers; the
sqlite-vec-specific tests move to a new tests_py/test_doctor_sqlite_vector_search.py
to keep test_doctor.py under 300 lines.

tests_py/invariants/test_I2_canonical_writer.py's ALLOWED_WRITERS line
numbers for sqlite_store.py shifted by +3 from this diff; updated to
match (same three writers, confirmed by reading each site, not assumed).

ADR renumbered 1087 -> 1089 (this branch's own number; sibling PRs #638
and #639 already claim 1088/1090) and rewritten around the actual
defect ('neural' stamped without its vector) instead of only the
install-list gap; the wiki ADR is now this fix's single source of truth
for the 'fallback' vs 'neural' semantics, referenced by every call site
that must not re-collapse them.

Left unaddressed, flagged for the maintainer: _try_load_vec logs INFO
and sets has_vec=False when the extension fails to load. Hardening it
to raise at store-open time would change startup behavior on
interpreters without loadable-extension support (common on macOS
system/Framework Python) -- a product decision, not bundled here.

Full suite: 9121 passed, 11 skipped (darwin, same pre-existing 11).
ruff and the craftsmanship gate both clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

* fix(consolidation): declare has_vec as a property on _FallbackWorklistStore

A bare has_vec: bool attribute annotation on the Protocol made pyright
flag SqliteMemoryStore as unsafely overlapping the protocol (its own
has_vec is a real @Property, not a data attribute), caught only once
CI ran pull_request-triggered checks on this branch for the first time.
Declaring has_vec as a Protocol @Property matches the real
implementation and resolves the structural-overlap false positive.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

* fix(sqlite): downgrade to 'fallback' instead of skipping the stamp

Review (PR #637) caught a real regression, worse than the bug this
branch fixes: the gate `model != "neural" or persisted` skipped
_stamp_embedding_model entirely when a neural encode had no persisted
vector. The row's embedding_model stayed at the schema default '',
which select_fallback_embeddings's own WHERE clause excludes -- the
row became permanently invisible to the re-embed worklist, unlike the
pre-fix bug (a plain 'neural' mislabel) that this same PR's vec-gap
self-heal sweep already catches.

Every insert/compression-update now always stamps something: a neural
encode with no persisted vector downgrades to 'fallback' (recoverable,
matches issue #169's unconditional-fallback contract); every other
case stamps its own resolved model name. Updated the test that
asserted the orphaned '' state as correct to assert the downgrade and
that the row surfaces in the worklist instead. ADR-1089 and its docs
mirror updated to describe the actual final behavior.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

* fix(tests): skip the neural-exclusion worklist test without sqlite-vec

CI caught this on all four Test (Python 3.1x) matrix jobs, which don't
install sqlite-vec (unlike the SQLite-backend job, which does):
test_select_fallback_embeddings_worklist asserted a 'neural'-tagged
insert is excluded from the fallback worklist, without has_vec being
true. That assumption only ever held by accident, on the pre-fix code:
a neural encode with no persisted vector used to be left unstamped
(''), which is invisible to the worklist query for a different reason
(the orphan bug this branch's own review just fixed). With the vector
never actually able to persist, this test's "neural one" now correctly
downgrades to 'fallback' and belongs in the worklist -- exactly
test_neural_stamp_downgrades_to_fallback_without_a_persisted_vector's
contract, already covered elsewhere in this file. Marked @_needs_vec,
matching the file's existing convention for assertions that are only
meaningful when sqlite-vec is actually available.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MyRfKj1vpBm2yx3sXCN8ik

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade 3.x → 4.x leaves the plugin on PostgreSQL: backend.json is never written when setup.py fails

1 participant