Skip to content

Commit 1d5a534

Browse files
cdeustclaude
andauthored
docs: describe recall's real fusion; stop storing layer file counts (#562, #563) (#566)
#562. The docs, the recall tool description and the MCP prompt called the read path "6-signal WRRF" (vector, BM25, heat, Hopfield, HDC, SR). The code fuses five signals inside the database: vector, full-text (ts_rank_cd / FTS5), trigram, effective heat and recency. PostgreSQL combines them as a weighted sum of max-normalised scores (infrastructure/pg_schema.py, recall_memories()); SQLite uses rank-based WRRF without the trigram signal (infrastructure/sqlite_store_search.py). Hopfield and HDC run after fusion (core/pg_recall_stages.py). Corrected in docs/mcp-tools.md, docs/architecture.md, docs/api-reference.md, docs/data-flow.md (diagram and signal tables), README.md, the recall handler's description strings and mcp_prompts.py. ADR-013 keeps its original design text with a dated note, written in the canonical wiki page and mirrored by scripts/check_project_wiki.py --write. #563. docs/module-inventory.md stored four per-layer file counts that had drifted from the tree. A count in this page changes with every new module and no gate can hold it exactly without conflicting across parallel branches, so the page now gives the measurement command instead. Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent a3b3a79 commit 1d5a534

10 files changed

Lines changed: 72 additions & 107 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -217,8 +217,9 @@ Recall@10 is the share of questions whose answer-bearing session appears in the
217217
retrieved sessions. MRR (mean reciprocal rank) rewards finding that session near the top. These
218218
numbers describe retrieval only; they do not measure whether an LLM writes a correct answer.
219219

220-
Retrieval fuses five signals through weighted reciprocal-rank fusion, then reranks with a
221-
cross-encoder: vector similarity, full-text search, trigram match, heat and recency. LoCoMo
220+
Retrieval fuses vector similarity, full-text search, trigram match, heat and recency, then
221+
reranks with a cross-encoder. PostgreSQL combines the signals as a weighted sum; SQLite uses
222+
weighted reciprocal-rank fusion and has no trigram signal. LoCoMo
222223
and BEAM results, the ablations and the floor gates are in [benchmarks/](benchmarks/).
223224

224225
## Storage

‎docs/adr/013-zikkaron-thermodynamic-memory.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ Implement a thermodynamic memory architecture in three tiers, each building on t
3232
### Tier 2 -- Navigation & Retrieval
3333

3434
- **6-signal WRRF fusion**: vector similarity + FTS5 BM25 + thermodynamic heat + Hopfield associative + HDC hyperdimensional + Successor Representation co-access
35+
- *Note, 2026-09-15 (#562):* this is the original design. The shipped read path fuses five signals inside the database (vector, full-text, trigram, heat, recency): a weighted sum of max-normalised scores on PostgreSQL (`recall_memories()` in `mcp_server/infrastructure/pg_schema.py`), rank-based WRRF on SQLite without the trigram signal (`mcp_server/infrastructure/sqlite_store_search.py`). Hopfield and HDC run after fusion (`mcp_server/core/pg_recall_stages.py`); Successor Representation is not on the `recall` path.
3536
- **HDC encoder** (`hdc_encoder.py`): 1024-dimensional bipolar hypervectors for content-addressable similarity
3637
- **Cognitive map** (`cognitive_map.py`): Successor Representation transition matrix from co-access patterns + 2D eigendecomposition projection
3738
- **Fractal hierarchy** (`fractal.py`): L0/L1/L2 hierarchical clustering for multi-scale recall

‎docs/api-reference.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ Store a memory through the predictive coding write gate.
142142

143143
### `recall`
144144

145-
Retrieve memories using 6-signal WRRF fusion (vector + FTS5 + heat + Hopfield + HDC + SR).
145+
Retrieve memories by fusing five signals server-side (vector, full-text, trigram, heat, recency; the SQLite backend has no trigram signal), then Hopfield, HDC and spreading-activation reranks, then FlashRank.
146146

147147
**Input Schema:**
148148

‎docs/architecture.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,7 @@ for current file counts per layer.
134134

135135
| Module | Purpose |
136136
|---|---|
137-
| `query_router.py` | Intent classification (temporal/causal/semantic/entity) + 6-signal WRRF fusion |
137+
| `query_router.py` | Deprecated shim (ADR-0234); intent now comes from `query_intent.py`, signal weights from `pg_recall_weights.py` |
138138
| `hdc_encoder.py` | 1024D bipolar hyperdimensional computing (bind/bundle/permute/similarity) |
139139
| `cognitive_map.py` | Successor Representation co-access graph + 2D projection |
140140
| `hopfield.py` | Hopfield network for content-addressable recall |
@@ -204,7 +204,7 @@ for current file counts per layer.
204204
| Module | Purpose |
205205
|---|---|
206206
| `remember.py` | Store memory through 4-signal predictive coding gate |
207-
| `recall.py` | Retrieve memories via 6-signal WRRF fusion |
207+
| `recall.py` | Retrieve memories: 5-signal server-side fusion, post-fusion reranks, FlashRank |
208208
| `consolidate.py` | Run maintenance: decay, compression, CLS, sleep compute |
209209
| `checkpoint.py` | Save/restore working state for hippocampal replay |
210210
| `narrative.py` | Generate project narrative from memories |

‎docs/data-flow.md‎

Lines changed: 39 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -53,46 +53,56 @@ For any language not listed, structural error markers still work, and the univer
5353

5454
## Memory Read Path (`recall`)
5555

56-
The read path uses intent-aware routing and multi-signal fusion.
56+
The read path classifies the query's intent, fuses five signals inside the
57+
database, then reranks. Paths below are relative to `mcp_server/`.
5758

5859
```mermaid
5960
flowchart TD
60-
Q[recall query] --> QE[Query Enrichment]
61-
QE -->|Doc2Query| QE1[Synthetic sub-questions]
62-
QE -->|Concept expansion| QE2[Synonym terms]
63-
QE1 & QE2 --> R[Query Router]
64-
R -->|classify intent| S{Intent Type}
65-
S -->|temporal| W1[Boost time signals]
66-
S -->|causal| W2[Boost causal graph]
67-
S -->|semantic| W3[Boost vector similarity]
68-
S -->|entity| W4[Boost entity graph]
69-
W1 & W2 & W3 & W4 --> F[6-Signal WRRF Fusion]
70-
F --> F1[Signal 1: Vector similarity]
71-
F --> F2[Signal 2: FTS5 full-text]
72-
F --> F3[Signal 3: Thermodynamic heat]
73-
F --> F4[Signal 4: Hopfield associative]
74-
F --> F5[Signal 5: HDC hyperdimensional]
75-
F --> F6[Signal 6: SR co-access]
76-
F1 & F2 & F3 & F4 & F5 & F6 --> RR[Reciprocal Rank Fusion]
77-
RR --> MR[Neuro-symbolic Rules]
78-
MR --> RES[Ranked Results]
79-
80-
style R fill:#d946ef,color:#000
61+
Q[recall query] --> X{ADR-NNNN or memory id?}
62+
X -->|yes| EX[Exact lookup, no fusion]
63+
X -->|no| I[Intent classification, regex]
64+
I --> W[Per-intent weights: fts, heat, ngram, recency]
65+
W --> F[5-signal fusion in recall_memories]
66+
F --> F1[Vector cosine, pgvector]
67+
F --> F2[Full-text, ts_rank_cd or FTS5]
68+
F --> F3[Trigram, PostgreSQL only]
69+
F --> F4[Effective heat]
70+
F --> F5[Recency]
71+
F1 & F2 & F3 & F4 & F5 --> RC[Recollection reranks, RRF k = 60]
72+
RC --> CE[FlashRank cross-encoder]
73+
CE --> MR[Nudges, neuro-symbolic rules, ordering]
74+
MR --> RES[Ranked results]
75+
76+
style I fill:#d946ef,color:#000
8177
style F fill:#06b6d4,color:#000
8278
style RES fill:#22c55e,color:#000
83-
style QE fill:#f59e0b,color:#000
8479
```
8580

81+
On PostgreSQL, `recall_memories()` combines the five signals as a weighted
82+
sum of max-normalised scores (`infrastructure/pg_schema.py`). On SQLite, the
83+
fusion is rank-based, w/(k + rank), over vector, FTS5, heat and recency, with
84+
no trigram signal (`infrastructure/sqlite_store_search.py`).
85+
8686
### Retrieval Signals
8787

88+
Fused inside the database:
89+
8890
| Signal | Module | What it measures |
8991
|---|---|---|
90-
| Vector similarity | `embedding_engine.py` | Cosine similarity between query and memory embeddings |
91-
| FTS5 full-text | `memory_store.py` | SQLite FTS5 BM25 ranking |
92-
| Thermodynamic heat | `thermodynamics.py` | Current heat value (recency + importance) |
93-
| Hopfield associative | `hopfield.py` | Content-addressable recall via energy minimization |
94-
| HDC hyperdimensional | `hdc_encoder.py` | 1024D bipolar hypervector similarity |
95-
| SR co-access | `cognitive_map.py` | Successor Representation transition probabilities |
92+
| Vector similarity | `infrastructure/pg_schema.py` | pgvector cosine distance between query and memory embeddings |
93+
| Full-text | `infrastructure/pg_schema.py` | `ts_rank_cd` on PostgreSQL, FTS5 on SQLite |
94+
| Trigram | `infrastructure/pg_schema.py` | `pg_trgm` similarity (PostgreSQL only) |
95+
| Thermodynamic heat | `infrastructure/pg_schema.py` | `effective_heat()`, decay computed at read time |
96+
| Recency | `infrastructure/pg_schema.py` | exponential decay on the memory's age |
97+
98+
Applied after fusion, each blended by RRF k = 60 (`core/pg_recall_stages.py`,
99+
`core/recall_pipeline.py`):
100+
101+
| Rerank | Module | What it measures |
102+
|---|---|---|
103+
| Hopfield completion | `core/hopfield.py` | Content-addressable recall via energy minimization |
104+
| HDC | `core/hdc_encoder.py` | 1024D bipolar hypervector similarity |
105+
| Spreading activation | `core/spreading_activation.py` | Entity-graph expansion, when `sa_mode` is `augment` |
96106

97107
### Intent Classification
98108

‎docs/mcp-tools.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ upstream MCP server is configured (55 total with both present).
2626
| `record_session_end` | Incremental profile update + session critique | <200ms |
2727
| `explore_features` | Interpretability exploration (features, attribution, persona, crosscoder) | <100ms |
2828
| `remember` | Store a memory through the 4-signal predictive coding gate | <100ms |
29-
| `recall` | Retrieve memories via 6-signal WRRF fusion | <200ms |
29+
| `recall` | Retrieve memories: 5-signal server-side fusion, post-fusion reranks, FlashRank | <200ms |
3030
| `consolidate` | Run maintenance: decay, compression, CLS, sleep compute | <5s |
3131
| `checkpoint` | Save/restore working state for hippocampal replay | <100ms |
3232
| `narrative` | Generate project narrative from stored memories | <500ms |
@@ -139,9 +139,10 @@ table above — the catalogue and the command ship in the same commit.
139139

140140
1. **Route**: Intent classification (temporal/causal/semantic/entity/knowledge_update/multi_hop)
141141
2. **Enrich**: Doc2Query expansion + concept synonyms
142-
3. **Fuse**: PL/pgSQL `recall_memories()` — WRRF fusion of vector + FTS + trigram + heat + recency (server-side)
143-
4. **Rerank**: FlashRank cross-encoder (client-side, top-3x candidates)
144-
5. **Filter**: Neuro-symbolic rules → ranked results
142+
3. **Fuse**: PL/pgSQL `recall_memories()` fuses vector, FTS (`ts_rank_cd`), trigram, heat and recency server-side as a weighted sum of max-normalised scores. The SQLite backend fuses vector, FTS5, heat and recency by rank, w/(k + rank), with no trigram signal
143+
4. **Recollect**: Hopfield completion, HDC, optional spreading activation, dendritic, emotional and mood reranks, reconsolidation, each blended by RRF k = 60
144+
5. **Rerank**: FlashRank cross-encoder (client-side, top-3x candidates)
145+
6. **Filter**: Neuro-symbolic rules → ranked results
145146

146147
### Cognitive Profile Pipeline
147148

‎docs/module-inventory.md‎

Lines changed: 11 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -6,71 +6,21 @@ description: "Per-layer module inventory for mcp_server/ — extracted from CLAU
66

77
Layer-by-layer catalogue of `mcp_server/`. This file is the curated/documented
88
subset that existed in CLAUDE.md before the #114 refactor — every module below
9-
was already described there; nothing has been dropped. Counts below are
10-
**measured**, not carried over from the prior (stale) prose, per the
11-
zetetic source rule:
9+
was already described there; nothing has been dropped.
10+
11+
Per-layer file counts are not recorded here. They change with every module
12+
added, split or moved, and a count written into this page drifts without any
13+
gate noticing: the four counts it carried until issue #563 no longer matched
14+
the tree. Measure a layer when you need the number:
1215

1316
```
14-
# source: measured on 2026-07-29 via
15-
# find mcp_server/<layer> -name "*.py" | grep -v __init__ | grep -v __pycache__ | wc -l
16-
# infrastructure/ re-measured 2026-07-30 (issue #275 split added 4 modules,
17-
# net file count +4; no infrastructure/ files were deleted)
18-
# core/ re-measured 2026-07-30 (issue #228 behaviour-preserving split of
19-
# core/context_assembly/condensers.py, over the 300-line §4.1 cap, into
20-
# condense_text/condense_code/condense_structured/condense_dispatch/
21-
# condense_stage.py + a thin re-export facade; net file count +5, no
22-
# core/ files were deleted)
23-
# core/ re-measured 2026-07-30 (issue #201: core/context_assembly/
24-
# active_retrieval.py deleted — no call site in production or benchmark
25-
# code, none in git history, none in any unmerged branch; net file
26-
# count -1)
27-
# shared/+core/+infrastructure/+handlers/ all re-measured 2026-08-10
28-
# (issue #406: temporal.py/temporal_normalize.py/temporal_timezones.py/
29-
# near_dup_calibration.py/write_class.py moved core/ -> shared/, all
30-
# five stdlib-only with infrastructure/ call sites that could not
31-
# legally import core/ -- shared/ +5, core/ -5)
32-
# infrastructure/ re-measured 2026-08-10 (issue: pg_store.py, 1384 lines,
33-
# split into 16 Pg*Mixin modules behind a thin facade, net +15; issue
34-
# #407: pg_store_auxiliary.py, 397 lines, split into 6 modules and
35-
# deleted, net +5; pg_store_queries.py, 401 lines, split into 3
36-
# modules, net +2; pg_store_stats.py, 406 lines, split into 3 modules,
37-
# net +2; pg_store_search.py, pushed to 301 lines by the #399
38-
# trust-term port (review-caught, PR #409 round 2), split into 2
39-
# modules, net +1 -- infrastructure/ net +25 from the pre-split
40-
# baseline of 93; the core/ and handlers/ deltas beyond the #406
41-
# move are unrelated drift accumulated upstream, not attributable to
42-
# either change)
43-
# shared/+core/+infrastructure/ re-measured 2026-08-10 (issue: PR #409
44-
# review round 3 rejected "declared violation" as an acceptance
45-
# criterion for infrastructure/wiki_store.py + wiki_schema_reader.py
46-
# importing core/ — the real defect was wiki-generation domain logic
47-
# mixed with storage access, not a misdirected import. Fix: the
48-
# 8-module pure wiki_pages/wiki_frontmatter/wiki_layout/wiki_readme/
49-
# wiki_index/wiki_page_builders/wiki_frontmatter_validation/
50-
# wiki_schema_loader cluster moved core/ -> shared/ (stdlib-only,
51-
# zero I/O, same #406 rationale); the one non-pure dependency
52-
# (core.wiki_sync.build_from_memory, real classifier judgment) is now
53-
# called from a new composition root, handlers/wiki_memory_sync.py,
54-
# which wires it to wiki_store.py's write primitives — infrastructure/
55-
# no longer imports core/ anywhere. wiki_store.py (439 lines,
56-
# pre-existing debt surfaced by this fix) also split under the
57-
# 300-line cap into wiki_pages_listing.py + wiki_reindex_io.py.
58-
# shared/ +8, core/ -8, infrastructure/ +2 (wiki_pages_listing.py,
59-
# wiki_reindex_io.py), handlers/ +1 (wiki_memory_sync.py, a
60-
# composition-root helper, not a registered tool)
61-
shared/ 39 files (23 documented below — curated subset)
62-
core/ 220 files (~87 documented below — curated subset, incl. core/streaming, core/context_assembly)
63-
infrastructure/ 120 files (52 documented below — curated subset)
64-
handlers/ 139 files (55 registered tools — see docs/mcp-tools.md — + composition-root helpers)
17+
find mcp_server/<layer> -name "*.py" ! -name "__init__.py" ! -path "*__pycache__*" | wc -l
6518
```
6619

67-
The prior CLAUDE.md text asserted "108 modules" for `core/` and similar
68-
counts for the other layers; that was a session-authored estimate, not a
69-
measured one, and had drifted from the actual tree. The counts above are the
70-
corrected, sourced figures. The module descriptions that follow remain a
71-
**curated subset** (the modules judged worth a one-line description at the
72-
time they were documented) — not a 1:1 listing of every file in the layer.
73-
Treat gaps as "undocumented," not "does not exist."
20+
The module descriptions that follow are a **curated subset** (the modules
21+
judged worth a one-line description when they were documented), not a 1:1
22+
listing of every file in the layer. Treat gaps as "undocumented," not "does
23+
not exist." The registered tools are listed in `docs/mcp-tools.md`.
7424

7525
## Dependency Rules
7626

‎mcp_server/handlers/recall.py‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
33
Composition root wiring infrastructure to core retrieval logic.
44
5-
Base retrieval uses pg_recall (intent-adaptive PG WRRF + FlashRank reranking).
5+
Base retrieval uses pg_recall (intent-adaptive 5-signal fusion + FlashRank reranking).
66
Production enrichments layer on top: prospective memory injection,
77
co-activation Hebbian learning, neuro-symbolic rules, strategic ordering.
88
"""
@@ -179,8 +179,9 @@
179179
},
180180
"description": (
181181
"Retrieve memories from the Cortex store using intent-adaptive PG "
182-
"recall (server-side WRRF fusion of vector + FTS + trigram + heat + "
183-
"recency) followed by FlashRank cross-encoder reranking and "
182+
"recall (server-side fusion of vector + FTS + trigram + heat + recency: "
183+
"a weighted sum on PostgreSQL, rank-based WRRF on SQLite) followed "
184+
"by FlashRank cross-encoder reranking and "
184185
"production enrichments (prospective memory injection, Hebbian "
185186
"co-activation strengthening, neuro-symbolic rules, strategic ordering "
186187
"to mitigate Lost-in-the-Middle, Liu et al. 2023). Use this before "
@@ -209,7 +210,7 @@
209210
"description": (
210211
"Natural-language query describing what to retrieve. Free "
211212
"text; intent (temporal/causal/semantic/entity/multi-hop) "
212-
"is auto-classified to weight the WRRF signals."
213+
"is auto-classified to weight the fusion signals."
213214
),
214215
"examples": [
215216
"why did we choose pgvector over Pinecone?",
@@ -292,7 +293,7 @@
292293
"description": (
293294
# source: ADR-0430
294295
"Opt in to cross-domain memories in the spreading-activation "
295-
"entity-graph expansion stage only. The primary WRRF search "
296+
"entity-graph expansion stage only. The primary fused search "
296297
"remains scoped to ``domain``. When true, expansion may inject "
297298
"memories from other domains reachable through shared entities. "
298299
"Defaults to false. Orthogonal to ``sa_mode``."
@@ -317,7 +318,7 @@
317318
"items": {"type": "string"},
318319
"description": (
319320
"Positive tag filter (OR): keep only memories that carry "
320-
"at least one of the listed tags. Applied after the WRRF "
321+
"at least one of the listed tags. Applied after the fused "
321322
"recall pipeline, at the same stage as the low-signal "
322323
'filter. Pass ``tags_any=["archival"]`` to retrieve only '
323324
"archival-tier memories."
@@ -330,7 +331,7 @@
330331
"items": {"type": "string"},
331332
"description": (
332333
"Positive tag filter (AND): keep only memories that carry "
333-
"ALL of the listed tags. Applied after the WRRF recall "
334+
"ALL of the listed tags. Applied after the fused recall "
334335
"pipeline, at the same stage as the low-signal filter."
335336
),
336337
"default": [],

‎mcp_server/mcp_prompts.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ class _Spec:
4242
),
4343
_Step(
4444
"recall",
45-
"Pull memories relevant to your focus via the 6-signal WRRF fusion.",
45+
"Pull memories relevant to your focus via recall's 5-signal fusion.",
4646
),
4747
_Step(
4848
"unified_search",

‎wiki/adr/cortex/0013-zikkaron-thermodynamic-memory.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ Implement a thermodynamic memory architecture in three tiers, each building on t
3030
### Tier 2 -- Navigation & Retrieval
3131

3232
- **6-signal WRRF fusion**: vector similarity + FTS5 BM25 + thermodynamic heat + Hopfield associative + HDC hyperdimensional + Successor Representation co-access
33+
- *Note, 2026-09-15 (#562):* this is the original design. The shipped read path fuses five signals inside the database (vector, full-text, trigram, heat, recency): a weighted sum of max-normalised scores on PostgreSQL (`recall_memories()` in `mcp_server/infrastructure/pg_schema.py`), rank-based WRRF on SQLite without the trigram signal (`mcp_server/infrastructure/sqlite_store_search.py`). Hopfield and HDC run after fusion (`mcp_server/core/pg_recall_stages.py`); Successor Representation is not on the `recall` path.
3334
- **HDC encoder** (`hdc_encoder.py`): 1024-dimensional bipolar hypervectors for content-addressable similarity
3435
- **Cognitive map** (`cognitive_map.py`): Successor Representation transition matrix from co-access patterns + 2D eigendecomposition projection
3536
- **Fractal hierarchy** (`fractal.py`): L0/L1/L2 hierarchical clustering for multi-scale recall

0 commit comments

Comments
 (0)