fix(storage): mark beliefs/summaries stale when source memory deleted - #1865

What this changes

Deleting a memory now updates the beliefs and consolidation summaries built from it, in the same transaction that already deletes the graph edges.

Why

Graph cleanup removed edges that touch a deleted hash and left everything derived from it in place. derived_from, contradicted_by and the dedup copy in metadata.merged_from_hashes kept the hash, and compression and association memories kept it in source_memory_hashes. Retention deletes memories on the normal path, so get_beliefs kept returning a belief whose evidence was gone.

Fixes #1857.

How it was verified

On main (a2ee57d3), with src/ swapped back, scripts/ci/check_tests_prove_fix.sh origin/main ran the new file and reported PASS - the changed tests fail without the change (pytest exit 1):

.venv/bin/pytest tests/consolidation/test_delete_stale_beliefs.py -q --tb=line
15 failed

Eight of those drive real sqlite-vec storage. They fail because derived_from still contains the deleted hash (delete, delete_memory, delete_by_tag, delete_by_tags, delete_by_timeframe, and the remote tombstone), and because _update_belief unions a tombstoned hash back out of merged_from_hashes. The other seven are the pure-function tests; on main they error with ModuleNotFoundError because storage/source_retraction.py does not exist yet.

On this branch:

.venv/bin/pytest tests/consolidation/test_delete_stale_beliefs.py tests/test_belief_dedup_producer.py tests/test_belief_connection_lock.py tests/consolidation/test_dead_storage_calls.py -q
67 passed

The new file alone is 15 passed. bash scripts/pr/pre_pr_check.sh was not run. The complexity and security half of that gate needs the local model, which is not available in this environment.

Changelog fragment

changelog.d/1857.fixed.md

Notes for the reviewer

Status name: ungrounded. A belief with nothing left in derived_from and contradicted_by gets that status, so the default get_beliefs(status='active') stops returning it. superseded stays what dedup uses for a loser. Confidence is not touched; challenge_belief still recalculates it.

The note on the issue suggests leaving that status flip for the next consolidation run, so a re-ingest under a new hash does not throw the belief away while consolidation is still catching up. This flips it at delete time. Otherwise the belief stays active with an empty source list, which is the bug. The row is not deleted, and a later challenge_belief can promote it again. merged_from_hashes is stripped in the same write, and _update_belief skips a merged hash that is no longer a live memory, so the next derivation cannot union the deleted hash back. When the memories table cannot be asked (the belief-only test databases), every hash is kept.

Summaries are flagged with metadata.stale_sources and not rewritten. source_memory_hashes still lists the deleted hash. Only memories tagged association, or both cluster and compressed, are touched. A user note that happens to store source_memory_hashes is left alone.

stale_sources is {count, updated_at}. On a belief, count accumulates as hashes are removed from the lists. On a summary, count is how many of the recorded sources are not live, so flagging the same gap twice does not increment it.

Lookup is json_each over the JSON columns, guarded with json_valid, against a temp table of the hashes just deleted. A SQLite without JSON functions falls back to a scan. On an in-memory store of 4000 beliefs, that lookup returned 240 citing rows in about 2.7 ms, so there is no belief_sources table.

The same helper runs from delete(), delete_memory() (the retention/forgetting path), delete_by_tag, delete_by_tags, delete_by_timeframe / delete_before_date (they call delete()), and the remote tombstone path, which already deletes graph edges in that transaction.

#1580 (evidence on inferred edges) is untouched.

The issue names draft #1714 as a read-time filter on get_beliefs. The PR open at that number today is the store-scope change, not that filter, and there is no read-time filter in tree to reconcile with. If one lands later, ungrounded already hides beliefs this path has cleaned. A read-time check would still matter for rows deleted before this change.

  

Confidence Score: 5/5

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Local or peer delete] --> B[Write tombstone]
  B --> C[Remove deleted belief evidence]
  C --> D[Flag affected summaries]
  D --> E[Record enabled sync events]
  E --> F[Commit together]
  C -->|Failure| G[Roll back and allow retry]
  D -->|Failure| G
  E -->|Failure| G

_{Reviews (8) · Last reviewed commit: "Merge branch 'main' into fix/delete-stal..." · Reviewed by Greptile}