Repository navigation
Expand file tree
/
Copy pathmain.tex
More file actions
2563 lines (2190 loc) · 110 KB
/
Copy pathmain.tex
File metadata and controls
2563 lines (2190 loc) · 110 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{xcolor}
% \graynoise: marks a per-ability table row whose delta is below the
% single-question noise floor of its slice (see top of Section 5).
\newcommand{\graynoise}[1]{\textcolor{gray}{\textit{#1}}}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{natbib}
\usepackage[margin=1in]{geometry}
\usepackage{hyperref}
% Utility commands
\newcommand{\ie}{i.e.}
\newcommand{\eg}{e.g.}
\newcommand{\cf}{cf.}
\title{Stage-Aware Context Assembly\\for Long-Context Memory Retrieval}
\author{
Clement Deust\\
Independent Researcher\\
\texttt{github.com/cdeust/Cortex}
}
\date{May 2026}
\begin{document}
\maketitle
\begin{abstract}
Dense vector retrieval degrades structurally at scale.
On BEAM \citep{Tavakoli2026}, the hardest publicly available
long-context memory benchmark at 10 million tokens per conversation,
a production-grade 5-signal fusion pipeline with cross-encoder
reranking drops from 0.591 MRR at 100K tokens to 0.353 MRR at
10M tokens---a 40\% degradation that no amount of reranking or
embedding model upgrade can address because the failure is geometric,
not parametric.
We present a structured context assembly architecture that recovers
this degradation: a two-primitive system combining
(1)~a priority-budgeted prompt decomposer with domain-aware
condensers and truncation warning injection, and
(2)~a stage-aware assembler (two phases implemented and evaluated;
a third summary-fallback phase is designed but not yet benchmarked) with submodular coverage
selection, Personalized PageRank entity graph traversal, and
schema-structured summary fallback.
On BEAM-10M (196 questions, 10 conversations), the assembler with
ground-truth stage boundaries (BEAM \texttt{plan\_id}) achieves 0.429
MRR, a $+21.5\%$ improvement over the flat baseline (0.353), with 8
of 10 memory abilities improving. Replacing the oracle labels with
\emph{label-free} temporal stage detection (day-level timestamp
partitioning, no metadata) raises this to 0.471 ($+33.4\%$):
temporal proximity proved a stronger stage signal than topic
boundaries on this benchmark
(\S\ref{sec:temporal-stage-results}), so the architecture deploys
without oracle metadata
(\S\ref{sec:stage-detection-deployment}).
The architecture was originally designed in September 2025 for
generating coherent product requirement documents on Apple
Intelligence's 4,096-token context window---one month before the
BEAM paper was published.
The key insight is that at scale, structured composition of what
enters context outperforms improving how individual items are
retrieved.
Code, data, and benchmark harness are publicly available.\footnote{\url{https://github.com/cdeust/Cortex}}
\end{abstract}
%----------------------------------------------------------------------
\section{Introduction}
\label{sec:intro}
Retrieval-augmented generation (RAG) has become the dominant paradigm
for grounding language models in external knowledge. The standard
recipe---embed documents, index them in a vector store, retrieve the
top-$k$ nearest neighbors at query time, and concatenate them into the
prompt---works well when the corpus is moderate and topically diverse.
But when applied to long-term conversational memory, where a single
user's history spans millions of tokens of thematically overlapping
content, this recipe fails in a specific and well-characterized way.
\subsection{The Scaling Failure}
\label{sec:scaling-failure}
Consider a memory system storing the conversational history of a
software developer over six months. At 100K tokens ($\sim$94 memories),
the content is sparse enough that a dense retrieval query returns
meaningfully different candidates. At 10M tokens ($\sim$7,500 memories),
the content has accumulated around a small number of recurring
topics---the same frameworks, the same debugging patterns, the same
architectural decisions discussed from slightly different angles
across hundreds of sessions. The embedding space becomes crowded
with near-identical vectors, and top-$k$ retrieval degenerates into
returning $k$ paraphrases of the same piece of information.
This is not a failure of any particular embedding model. Three
well-documented phenomena make it structural:
\paragraph{Hubness} \citep{Radovanovic2010}.
In high-dimensional spaces, certain points become disproportionately
frequent nearest neighbors of many queries. Formally, let $N_k(x)$
denote the number of times point $x$ appears in the $k$-nearest-neighbor
lists of all other points. In uniformly distributed data,
$\mathbb{E}[N_k(x)]$ is constant, but the variance of $N_k$ grows
with dimensionality. The skewness of the $N_k$ distribution---the
\emph{hubness}---increases monotonically with $d / \log n$ where $d$
is the dimensionality and $n$ is the sample size.
\citet{Radovanovic2010} showed that this is an inherent property of
the geometry, not an artifact of particular distance metrics. As
corpus size grows, a small set of ``hub'' memories dominates the
top-$k$ results regardless of query content. At 7,500 memories in
384 dimensions, we observe empirically that the top-5 retrievals for
topically distinct queries share 2--3 common memories---the hubs.
These hub memories are not necessarily irrelevant; they are typically
the most ``central'' memories that have high average similarity to
all queries. But their dominance crowds out the specific, targeted
memories that would actually answer the query.
\paragraph{Concentration of distances} \citep{Beyer1999}.
As dimensionality grows relative to sample size, the ratio of the
nearest-neighbor distance to the farthest-neighbor distance converges
to 1. Formally, for i.i.d.\ points drawn from any distribution with
finite variance:
\begin{equation}
\lim_{d \to \infty} \Pr\!\left[\frac{\|X_{\max} - q\| - \|X_{\min} - q\|}{\|X_{\min} - q\|} \leq \epsilon\right] = 1
\label{eq:concentration}
\end{equation}
for any $\epsilon > 0$. The practical consequence: in 384 dimensions
with 7,500 points, the gap between the ``most relevant'' and ``50th
most relevant'' memory shrinks to the point where cosine similarity
cannot reliably discriminate them. Adding more dimensions helps, but
the next phenomenon places a lower bound on how many dimensions suffice.
\paragraph{Dimensionality lower bounds} \citep{Larsen2017}.
The Johnson--Lindenstrauss lemma guarantees that pairwise distances
among $n$ points can be preserved to within $(1 \pm \epsilon)$ in
$O(\epsilon^{-2} \log n)$ dimensions. \citet{Larsen2017} proved
this is tight: any embedding that preserves pairwise distances among
$n$ points to within $(1 \pm \epsilon)$ requires
$\Omega(\epsilon^{-2} \log n)$ dimensions. For $n = 7{,}500$ and
$\epsilon = 0.1$ (10\% distance preservation), this gives:
\begin{equation}
d \geq \frac{1}{\epsilon^2} \cdot \ln(n) = \frac{1}{0.01} \cdot \ln(7500) \approx 100 \cdot 8.92 \approx 892
\label{eq:jl-bound}
\end{equation}
Our 384-dimensional embeddings (all-MiniLM-L6-v2) operate below
this lower bound, meaning pairwise distance preservation to 10\%
accuracy is not guaranteed even in principle for 7,500 points.
We note this is a lower bound on \emph{pairwise-distance} preservation,
not on nearest-neighbor \emph{ordering}, which can be preserved below
the JL bound; we cite it as indicative of the regime, not as a proof
that retrieval must fail at 384 dimensions. The empirical degradation
in Table~\ref{tab:10m} is the actual evidence.
The consequence is measurable. On the BEAM benchmark
\citep{Tavakoli2026}, our production Weighted Reciprocal Rank Fusion
(WRRF) pipeline---which fuses five PostgreSQL-side signals (vector
similarity, full-text search, trigram matching, thermodynamic heat,
temporal recency) with client-side FlashRank cross-encoder reranking---scores:
\begin{itemize}
\item \textbf{BEAM-100K} (94 memories/conversation): 0.591 MRR
\item \textbf{BEAM-10M} (7,500 memories/conversation): 0.353 MRR
\end{itemize}
A 40\% degradation at 100$\times$ scale. No reranking layer, query
rewriting strategy, or embedding model upgrade addresses a geometric
ceiling. The architecture must change.
\subsection{The Insight}
\label{sec:insight}
The insight behind this work is that the problem is not \emph{how} to
retrieve better, but \emph{what} to compose into context. A flat
retrieval pipeline treats the context window as a ranked list: the
top-$k$ most similar items, concatenated. A structured assembly
pipeline treats the context window as a \emph{document} with typed
sections, priority ordering, and explicit awareness of what was
included and what was cut.
This distinction matters most when the corpus is large and
thematically dense---precisely the regime where flat retrieval fails.
By partitioning the corpus into topical stages, retrieving within the
current stage first, following entity graph connections to related
stages, and falling back to summaries for uncovered stages, the
assembler recovers information that flat top-$k$ cannot reach.
The analogy is to how humans prepare for a meeting: you do not read
every email from the past year sorted by relevance score. You
identify the current topic, gather the directly relevant material,
follow references to related discussions, and glance at summaries of
background context. The structured approach is not better because
it uses a superior search algorithm---it is better because it
organizes the search into scopes that match the information need.
\subsection{Origin}
\label{sec:origin}
This architecture did not originate from a retrieval research project.
It was designed in September 2025 for a production Swift application
(\texttt{ai-architect-prd-builder}) that generates comprehensive
product requirement documents using Apple Intelligence, which has a
4,096-token context window. The application must produce coherent
9-page documents with cross-referenced requirements, working code
examples, verification reports, and Jira ticket specifications---none
of which fit in the model's context simultaneously.
The solution: \emph{do not try to fit everything. Structure what
goes in, prioritize it, and tell the model what was cut.}
This principle---born from a practical constraint on a tiny context
window---turns out to be exactly what is needed for 10-million-token
memory retrieval. The problem is structurally identical at both
scales: the available context is smaller than the relevant information,
so you must be intelligent about what enters the context.
\subsection{Contributions}
\label{sec:contributions}
We make four contributions:
\begin{enumerate}
\item \textbf{ContextDecomposer}: a priority-budgeted prompt
assembly primitive with typed placeholder slots, domain-aware
condensers, progressive condensation, and truncation warning
injection---a design contribution we found undescribed in the
2024--2026 retrieval, prompt engineering, or agent memory
literature we surveyed, and whose effect on answer quality we do
not evaluate here.
\item \textbf{StageAwareContextAssembler}: a hierarchical
retrieval architecture (two phases evaluated; third designed) with a configurable budget
split (default 60/30/10, where the 10\% is reserved for the
designed-but-unwired summary-fallback phase), combining submodular coverage selection
\citep{Krause2008}, Personalized PageRank entity graph traversal
\citep{Gutierrez2024}, and schema-structured summary fallback
\citep{Tse2007}.
\item \textbf{Empirical validation} on BEAM at two scales (100K
and 10M tokens), showing that structured assembly is net-flat at
small scale (where flat retrieval is already sufficient) and
provides a $+21.5\%$ improvement at large scale with oracle
stage labels, rising to $+33.4\%$ with label-free temporal stage
detection (where flat retrieval collapses)---with 8 of 10 memory
abilities improving in both configurations.
\item \textbf{Ablation analysis} isolating the contribution of
submodular selection ($+17.6\%$ over naive top-$k$) and
establishing that the architecture's value is compositional
rather than attributable to any single mechanism.
\end{enumerate}
\subsection{Paper Organization}
\label{sec:organization}
Section~\ref{sec:background} reviews related work across dense
retrieval, long-context memory systems, hierarchical retrieval, and
the biological foundations that informed design choices.
Section~\ref{sec:method} presents the method in detail, including
the ContextDecomposer algorithm, the three-phase assembler, and the
integration with the existing Cortex pipeline.
Section~\ref{sec:experiments} describes the experimental setup,
benchmarks, baselines, and measurement protocol.
Section~\ref{sec:results} presents results at both scales with
per-ability breakdowns. Section~\ref{sec:analysis} provides
analysis of scale-dependent behavior, the multi-session reasoning
sign flip, and comparison with published systems.
Section~\ref{sec:provenance} documents the provenance timeline with
verifiable commit SHAs. Section~\ref{sec:limitations} discusses
limitations and future work. Section~\ref{sec:conclusion} concludes.
%----------------------------------------------------------------------
\section{Background and Related Work}
\label{sec:background}
\subsection{Dense Retrieval at Scale}
\label{sec:dense-retrieval}
The standard dense retrieval pipeline embeds queries and documents
into a shared vector space and retrieves by nearest-neighbor search.
The bi-encoder architecture \citep{Karpukhin2020,Reimers2019} maps
queries and passages independently, enabling pre-computation of
passage embeddings and sublinear retrieval via approximate
nearest-neighbor indices (HNSW, IVF, product quantization).
\paragraph{Benchmarking dense retrieval.}
BEIR \citep{Thakur2021} established zero-shot evaluation across 18
diverse datasets, revealing that dense retrievers trained on MS~MARCO
generalize poorly to domain-specific corpora. MTEB
\citep{Muennighoff2023} extended this to 56 datasets across 8 tasks,
becoming the standard benchmark for embedding models. Both benchmarks
evaluate on corpora where documents are topically diverse---web pages,
Wikipedia passages, scientific abstracts, and forum posts.
\paragraph{Long-document retrieval.}
LongEmbed \citep{Zhu2024} extends evaluation to documents exceeding
8,000 tokens, testing whether embedding models preserve relevance
over longer passages. However, it still assumes a diverse corpus:
the challenge is encoding long documents, not discriminating among
thousands of topically similar documents from a single user.
\paragraph{The gap in existing benchmarks.}
The failure mode we address---topical density within a single user's
conversational history, where 7,500 memories discuss overlapping
topics in similar vocabulary---is not covered by existing retrieval
benchmarks. LongMemEval \citep{Wu2025} evaluates conversational
memory but at modest scale ($\sim$115K tokens, $\sim$40 sessions).
LoCoMo \citep{Maharana2024} tests multi-hop and adversarial queries
but caps at 10 conversations. BEAM is the first benchmark to
stress-test memory retrieval at 10M tokens per conversation, and
the results confirm that existing systems collapse at this scale:
the best published system (LIGHT with Llama-4-Maverick) achieves
only 0.266.
\subsection{Reciprocal Rank Fusion and Hybrid Search}
\label{sec:rrf}
Our baseline, Weighted Reciprocal Rank Fusion (WRRF), extends the
original Reciprocal Rank Fusion \citep{Cormack2009} with per-signal
weights. Given $m$ ranking signals, the fused score for document $d$
is:
\begin{equation}
\text{WRRF}(d) = \sum_{i=1}^{m} \frac{w_i}{k + r_i(d)}
\label{eq:wrrf}
\end{equation}
where $r_i(d)$ is the rank of $d$ under signal $i$, $k$ is a
smoothing constant (we use $k = 60$, following the original RRF
recommendation), and $w_i$ is the weight for signal $i$.
Cortex's WRRF pipeline fuses five signals:
\begin{table}[h]
\centering
\caption{WRRF signal configuration.}
\label{tab:wrrf-signals}
\begin{tabular}{lcc}
\toprule
Signal & Weight & Computed At \\
\midrule
Vector similarity (pgvector HNSW, 384D) & 0.35 & PostgreSQL \\
Full-text search (tsvector, BM25-weighted) & 0.25 & PostgreSQL \\
Trigram matching (pg\_trgm) & 0.15 & PostgreSQL \\
Thermodynamic heat (access recency/frequency) & 0.15 & PostgreSQL \\
Temporal recency (exponential decay) & 0.10 & PostgreSQL \\
\bottomrule
\end{tabular}
\end{table}
All five signals are computed server-side in a single PL/pgSQL stored
procedure (\texttt{recall\_memories()}), returning pre-fused results.
Client-side, FlashRank (ONNX cross-encoder) reranks the top-$3k$
candidates to produce the final ranking.
This pipeline is strong at moderate scale: 98.2\% R@10 on
LongMemEval (clean run, July 2026; code SHA \texttt{28145f0}) and
94.2\% R@10 on LoCoMo (E1 v3 run, May 2026, before the later plasticity fix; code SHA
\texttt{ef178da7}). The five-signal fusion
mitigates any single signal's weakness (\eg, vector similarity
misses lexical matches that trigram catches; FTS misses paraphrases
that vectors catch). But at BEAM-10M scale, all five signals suffer
from the same underlying geometric degradation, and fusion of
degraded signals produces a degraded result.
\subsection{Long-Context Memory Systems}
\label{sec:memory-systems}
Several systems have been proposed for persistent conversational
memory, each addressing different aspects of the long-term memory
challenge:
\paragraph{MemGPT / Letta} \citep{Packer2024}.
Implements a tiered memory system with conversation buffer (recent
messages), archival storage (compressed older messages), and recall
storage (searchable facts). Memory management is delegated to the
LLM via tool calls---the model decides when to read, write, and
search its own memory. This is an elegant design for self-managing
memory, but it relies on the LLM's judgment for retrieval,
introducing the same failure modes as any single-signal retrieval
(the LLM may generate poor search queries) plus additional latency
and cost for each memory operation. Letta does not implement
structured context assembly, priority budgeting, or entity graph
traversal.
\paragraph{mem0} \citep{Chheda2024}.
Extracts key-value facts from conversations using an LLM and stores
them as a flat memory bank. Retrieval is by embedding similarity
over the extracted facts, with optional filtering by category or time
range. The extraction step is valuable (it produces cleaner, more
searchable memories than raw conversation turns), but the retrieval
layer is standard dense search without any structural organization.
At scale, mem0 would face the same hubness and concentration issues
we document.
\paragraph{MIRIX} \citep{Wang2025}.
The closest architectural neighbor in the active retrieval dimension.
MIRIX introduces three key mechanisms: (1)~typed memory categories
(episodic, semantic, working), (2)~active retrieval---the agent
reformulates queries before searching, and (3)~reflection---the
agent periodically reviews and consolidates its memories. The
system reports 85.4\% on LoCoMo. However, MIRIX does not implement
stage-scoped retrieval, entity graph traversal, priority-budgeted
prompt assembly, or truncation awareness. Its active retrieval
component (query reformulation via LLM) is complementary to our
approach and could be composed with our assembler.
\paragraph{A-MEM} \citep{Xu2025}.
Introduces Zettelkasten-style agentic memory with on-write
reconsolidation: when a new memory is stored, the system uses an LLM
to compare it against existing memories, create bidirectional links,
and update an index. The write-time intelligence builds a richer
memory graph, which could benefit our Phase~2 entity traversal.
However, A-MEM's retrieval layer is standard dense search, and the
write-time LLM calls add significant latency and cost at scale.
\paragraph{LIGHT} \citep{Tavakoli2026}.
The strongest published system on BEAM, achieving 0.266 overall on
BEAM-10M with Llama-4-Maverick. LIGHT implements a three-tier
architecture: \emph{episodic memory} (stores raw conversation turns,
indexed for retrieval), \emph{working memory} (a fixed-size buffer
of the most recently accessed memories, refreshed each query), and
\emph{scratchpad} (an LLM-generated intermediate representation that
helps the reader reason across multiple retrieved passages).
LIGHT's three tiers serve different purposes than our three phases.
LIGHT's working memory is a recency-biased buffer; our Phase~1 is a
relevance-biased stage-scoped selection. LIGHT's scratchpad is a
reader-side reasoning aid; we have no equivalent. LIGHT does not
implement priority budgeting, truncation awareness, submodular
diversity, or entity graph traversal. Its scratchpad mechanism and
our context assembly are complementary approaches that could
potentially be combined.
\subsection{Hierarchical and Graph-Based Retrieval}
\label{sec:hierarchical}
\paragraph{HippoRAG} \citep{Gutierrez2024}.
Draws on the hippocampal indexing theory \citep{Teyler2007} to propose
a retrieval architecture that separates the neocortex (knowledge graph
of extracted entities and relations) from the hippocampus (passage
store) and bridges them via Personalized PageRank. The PPR walk
propagates relevance from query-mentioned entities outward through
the knowledge graph, scoring passages by the aggregated PPR mass of
their contained entities.
On multi-hop QA benchmarks (MuSiQue, 2WikiMultihopQA, HotpotQA),
HippoRAG demonstrates that graph-based traversal reaches relevant
passages that flat dense retrieval misses---particularly when the
answer requires synthesizing information from passages that do not
directly mention the query terms but are connected through shared
entities.
We adopt their PPR formulation for Phase~2 of our assembler but
apply it to a different problem: cross-stage entity bridging in
conversational memory rather than multi-hop factoid QA. The key
adaptation is the seeding strategy: HippoRAG seeds PPR on
query-extracted entities, while we seed on entities extracted from
Phase~1 results. This two-stage seeding (WRRF retrieval first, then
PPR traversal) is more robust than direct query seeding because
Phase~1 results are enriched with the memory system's entity
annotations, which are more complete than entities extractable from
a short query string.
\paragraph{RAPTOR} \citep{Sarthi2024}.
Builds a tree of summaries over document chunks using recursive LLM
summarization: leaf nodes are original chunks, and each internal node
summarizes its children. Retrieval can occur at any level of the
tree, enabling queries to match at different levels of abstraction.
The hierarchical summary structure is related to our Phase~3 summary
fallback, but RAPTOR operates on documents (where the tree structure
reflects the document's own hierarchy) rather than conversational
stages (where the structure reflects topical shifts over time).
RAPTOR's LLM summarization cost at build time is significant; we use
rule-based schema extraction instead.
\paragraph{Late Chunking} \citep{JinaAI2025}.
Proposes deferring chunking until after encoding with a long-context
model, so each chunk's embedding reflects the full document context.
This addresses a different failure mode---loss of cross-chunk context
during embedding---and is complementary to our approach. Late
chunking could improve the quality of per-stage embeddings within our
Phase~1.
\paragraph{ColBERT and Multi-Vector Retrieval} \citep{Khattab2020}.
Represents each passage as a set of token-level vectors and computes
relevance via MaxSim aggregation. This finer-grained representation
mitigates some hubness effects by allowing individual tokens to
contribute independently. However, the storage and computation
overhead (one vector per token) is substantial at 7,500-memory scale,
and the approach does not address the stage-composition problem.
\subsection{Submodular Optimization for Information Selection}
\label{sec:submodular-background}
Submodular set functions have a rich history in information selection
problems. A set function $f: 2^V \to \mathbb{R}$ is submodular if
for all $A \subseteq B \subseteq V$ and $e \notin B$:
\begin{equation}
f(A \cup \{e\}) - f(A) \geq f(B \cup \{e\}) - f(B)
\label{eq:submodular}
\end{equation}
This \emph{diminishing returns} property formalizes the intuition that
adding an item is more valuable when less has already been selected.
\paragraph{Sensor placement} \citep{Krause2008}.
Proved that for monotone submodular functions, the greedy algorithm
(iteratively select the item with the highest marginal gain) achieves
a $(1 - 1/e) \approx 0.63$ approximation to the optimal set. This
guarantee holds regardless of the constraint set (cardinality,
matroid, knapsack), with different approximation ratios for different
constraints.
\paragraph{Document summarization} \citep{Lin2011}.
Applied submodular optimization to extractive summarization, treating
sentence selection as a budget-constrained submodular maximization
problem. The objective combines coverage (how much of the document's
information is represented) with diversity (how different the selected
sentences are).
\paragraph{Maximal Marginal Relevance} \citep{Carbonell1998}.
The MMR criterion---$\text{score}(c) - \lambda \cdot \max_{s \in S}
\text{sim}(c, s)$---is the most common instantiation of submodular
selection in retrieval. \citet{Carbonell1998} introduced it for
reranking search results; we apply it to memory selection within a
stage. When $\lambda < 1$, the MMR objective is submodular (proof:
the max-similarity penalty is a modular function, and the difference
of a modular function from a monotone function is submodular).
\subsection{Biological Foundations}
\label{sec:bio-foundations}
The stage-aware assembly architecture has structural parallels to
established neuroscience models. We emphasize that these are design
analogies rather than mechanistic implementations---we do not claim
that the algorithms replicate biological processes, only that the
biological principles informed useful design choices.
\paragraph{Hippocampal indexing theory} \citep{Teyler2007}.
The hippocampus stores pointers to neocortical representations, not
the representations themselves. During retrieval, hippocampal
activation triggers reactivation of the corresponding neocortical
patterns. This ``index + content'' separation maps to our Phase~2
architecture: the entity graph serves as an index (hippocampus),
and the memories stored in PostgreSQL serve as the content
(neocortex). PPR traversal over the entity graph activates the
memories that share the most entity overlap with the current query
context.
\paragraph{Grid modules} \citep{Stensola2012}.
The entorhinal cortex organizes spatial information in modular grids
at different scales---small grids for local navigation, large grids
for global orientation. Our two-phase structure (with planned summary tier) (current stage,
adjacent stages, all stages) mirrors this multi-scale organization:
Phase~1 operates at the finest scale (within-stage), Phase~2 at an
intermediate scale (across nearby stages via entity connections), and
Phase~3 at the broadest scale (summaries of all stages). The analogy
should not be pushed further than the structural level---we do not
implement grid-like spatial representations.
\paragraph{Schema consolidation} \citep{Tse2007}.
Prior knowledge structures (schemas) facilitate rapid integration of
new information that is consistent with existing schemas.
\citet{Tse2007} showed that rats with a pre-existing spatial schema
could learn new object-location associations in a single trial,
compared to weeks without a schema. Our Phase~3 summary fallback
draws on this principle: schema-structured summaries of each stage
serve as compressed prior knowledge that new queries can be matched
against, even when the original memories are too numerous or too
distant to retrieve directly.
\paragraph{Complementary learning systems} \citep{McClelland1995}.
The hippocampal and neocortical systems serve complementary roles:
hippocampal learning is fast and episodic (one-shot, detail-rich),
while neocortical learning is slow and semantic (gradual,
pattern-extracting). Our architecture embodies this complementarity:
Phase~1 retrieves specific episodic memories (fast, detail-rich),
while Phase~3 retrieves consolidated semantic summaries (compressed,
pattern-level). Phase~2 bridges the two via entity-structural
connections.
%----------------------------------------------------------------------
\section{Method}
\label{sec:method}
The architecture comprises two core primitives that can be used
independently or composed. The \textbf{ContextDecomposer} manages
the prompt-level budget: given a template with typed placeholders, it
allocates tokens by priority and communicates truncation decisions to
the reader model. The \textbf{StageAwareContextAssembler} manages
the retrieval-level composition: given a query and a corpus
partitioned into stages, it selects memories from three scopes with
configurable budget proportions.
Figure~\ref{fig:dataflow} illustrates the conceptual data flow.
\begin{figure}[t]
\centering
\small
\begin{verbatim}
Query
|
v
[Stage Detection] --> current_stage
|
v
[Phase 1: Own-Stage WRRF + Submodular Select]
--> 60% budget
|
v
[Phase 2: Entity Extract --> PPR Walk
--> Cross-Stage Select] --> 30% budget
|
v
[Phase 3: Summary Fallback for Uncovered Stages]
--> 10% budget
|
v
[Structured Context:
## Current Stage | ## Related Prior
| ## Summaries]
|
v
[ContextDecomposer: Priority Budget +
Condense + Truncation Warning]
|
v
Final Prompt --> Reader Model
\end{verbatim}
\caption{Conceptual data flow of the two-primitive architecture. The
StageAwareContextAssembler produces structured context from three
retrieval phases; the ContextDecomposer fits it into the model's
context window with priority budgeting.}
\label{fig:dataflow}
\end{figure}
\subsection{ContextDecomposer: Priority-Budgeted Prompt Assembly}
\label{sec:decomposer}
A prompt template contains typed placeholder slots. Each slot is
defined by a 4-tuple:
\begin{equation}
P_i = (k_i, v_i, p_i, c_i)
\label{eq:placeholder}
\end{equation}
where $k_i$ is the template key (\eg, \texttt{\{\{CONTEXT\}\}},
\texttt{\{\{QUERY\}\}}, \texttt{\{\{RELATED\}\}}), $v_i$ is the
content to fill it, $p_i \in \mathbb{N}$ is the priority rank
(lower = more important; priority~1 is the last to be condensed),
and $c_i: (\text{text}, \text{budget}) \to \text{text}$ is an
optional domain-aware condenser function.
The assembly algorithm operates in six steps:
\subsubsection{Step 1: Shell Computation}
Compute the template with all placeholders replaced by empty strings.
Let $s = \tau(\text{shell})$ where $\tau$ is the token estimator
(default: conservative $\lceil |text| / 3 \rceil$, swappable for
tiktoken at integration).
\subsubsection{Step 2: Budget Allocation}
Given context window $W$ and headroom fraction $h$ (default 0.75,
reserving 25\% for the response):
\begin{equation}
B = \max\!\left(300, \lfloor W \cdot h \rfloor - s\right)
\label{eq:budget}
\end{equation}
The variable budget $B$ is the total token budget available for all
placeholder content. The minimum of 300 tokens ensures that even
with a large shell, some content always survives.
\subsubsection{Step 3: Fast Path}
If $\sum_i \tau(v_i) \leq B$, all placeholders are used verbatim.
No condensation is needed. This is the common case for short queries
with small context.
\subsubsection{Step 4: Progressive Condensation}
When the fast path fails, placeholders are condensed progressively
from least to most important. Sort placeholders by descending
priority number (highest $p_i$ first---least important first). For
each placeholder $P_i$ in this order:
\begin{enumerate}
\item Compute a proportional share of the remaining budget:
\begin{equation}
\text{share}_i = \max\!\left(50, \left\lfloor \frac{B_{\text{remaining}}}{|\{P_j : j \geq i,\; P_j \text{ not yet assigned}\}|}\right\rfloor\right)
\label{eq:share}
\end{equation}
\item If $\tau(v_i) \leq \text{share}_i$, use $v_i$ verbatim and
subtract $\tau(v_i)$ from $B_{\text{remaining}}$.
\item Otherwise, apply the condenser:
\begin{equation}
v_i' =
\begin{cases}
c_i(v_i, \text{share}_i) & \text{if } c_i \neq \text{null} \\
\text{truncate}(v_i, \text{share}_i) & \text{otherwise}
\end{cases}
\label{eq:condense}
\end{equation}
\item Use $v_i'$ and subtract $\min(\tau(v_i'), B_{\text{remaining}})$
from $B_{\text{remaining}}$.
\end{enumerate}
The proportional allocation ensures that low-priority placeholders
with small content are not penalized---they get their full content if
it fits within their share. Only placeholders that exceed their share
are condensed.
\subsubsection{Step 5: Post-Assembly Safety Loop}
After all placeholders are assigned, assemble the final prompt by
substituting each placeholder with its assigned content. If the
assembled prompt still exceeds $(W - \text{safety\_margin})$ tokens
(due to estimation errors or template overhead), iteratively halve
the content of the lowest-priority placeholder at the nearest line
boundary. This is a defensive measure---it should rarely trigger if
the budget computation is accurate.
The loop terminates after at most 50 iterations or when no further
reduction is possible.
\subsubsection{Step 6: Truncation Warning Injection}
For each placeholder whose surviving fraction is below 90\%:
\begin{equation}
\text{surviving}(P_i) = \frac{\tau(\text{final}_i)}{\tau(\text{original}_i)}
\label{eq:surviving}
\end{equation}
build a warning line. If any placeholders were materially truncated,
prepend a banner to the final prompt:
\begin{verbatim}
WARNING: CONTEXT TRUNCATION
The following sections were truncated to fit
the context window.
You may be missing information. Prioritize the
content you CAN see.
- {{RELATED_CONTEXT}}: 45% retained (340/756 tokens)
- {{STAGE_SUMMARIES}}: 22% retained (88/400 tokens)
\end{verbatim}
\paragraph{Design contribution.}
The truncation warning is, to our knowledge, undescribed in the
2024--2026 literature we surveyed; we present it as a design
contribution and do not evaluate its effect on answer quality here
(an A/B on truncated-evidence questions is the natural test and is
left to future work). We surveyed the
2024--2026 literature across RAG, prompt engineering, context
management, and agent architectures and found no published work we are aware of that combines these specific primitives
for injecting explicit truncation awareness into the prompt itself.
The closest mechanisms are: Anthropic's contextual retrieval (2024),
which adds chunk-level LLM summaries but enriches content rather than
communicating truncation decisions; LangChain/LlamaIndex token
budgeting, which truncates silently without informing the model what
was cut; and OpenAI's function calling with \texttt{max\_tokens}
constraints, which truncates tool outputs but does not inject a
warning about the truncation.
The design rationale is straightforward: a model that knows what it
\emph{cannot} see can hedge its confidence appropriately, request
clarification, or flag uncertainty. A model that receives a silently
truncated context has no basis for distinguishing ``this information
does not exist'' from ``this information was cut.'' We hypothesize
(but have not experimentally verified) that truncation warnings
improve answer quality for questions whose evidence was partially
truncated.
\subsubsection{Domain-Aware Condensers}
Generic truncation (keeping the first $N$ tokens) is a poor strategy
because the most important information is rarely at the beginning.
Domain-aware condensers exploit structural properties of different
content types to preserve high-signal content and drop filler:
\begin{table}[h]
\centering
\caption{Domain-aware condenser strategies.}
\label{tab:condensers}
\small
\begin{tabular}{lll}
\toprule
Content Type & Strategy & Signal Preserved \\
\midrule
User messages & First sentence + questions + last & Intent, queries \\
Assistant messages & Code blocks verbatim; prose to topics & Facts, implementations \\
Entity triples & Keep $(S, P, O)$ lines; drop prose & Relationship structure \\
Timeline events & Extract \texttt{[when] what} format & Temporal anchors \\
Code blocks & Function/class/import signatures only & API surface \\
\bottomrule
\end{tabular}
\end{table}
The condenser is selected automatically by content shape detection.
The detection heuristics are: (1)~tag-driven (highest priority): if
the memory has tags like \texttt{code}, \texttt{timeline},
\texttt{event}, dispatch directly; (2)~code fence detection: presence
of triple backticks or $\geq 3$ indented lines triggers the assistant
message condenser; (3)~arrow operator count: $\geq 2$ occurrences of
\texttt{->} triggers the entity triple condenser; (4)~role prefix
detection: lines starting with \texttt{[user]:} or
\texttt{[assistant]:} trigger the appropriate message condenser;
(5)~default: treat as prose user message.
Each condenser is a pure function
$(\text{text}, \text{token\_budget}) \to \text{text}$ with a fast
path: if the input already fits within the budget, return it unchanged.
\subsection{StageAwareContextAssembler}
\label{sec:assembler}
A \emph{stage} is a distinct topical segment of a conversation---analogous to a work session, a debugging episode, or a design
discussion. Stage detection is pluggable (Section~\ref{sec:stage-detection}).
The assembler operates in three phases with a configurable budget
split $(\beta_1, \beta_2, \beta_3)$ where
$\beta_1 + \beta_2 + \beta_3 = 1$ (default: 0.6, 0.3, 0.1).
Of the three phases, Phases~1--2 are implemented and evaluated in this
paper; Phase~3 (\S\ref{sec:phase3}) is designed and interface-complete
but returns empty in the evaluated build.
\subsubsection{Stage Detection}
\label{sec:stage-detection}
The \texttt{StageDetector} interface abstracts stage assignment with
two operations: \texttt{stage\_of(memory) -> str} (assigns a single
memory to its stage ID) and \texttt{all\_stages(corpus) -> list[str]}
(returns all distinct stages in a stable order).
We provide three implementations:
\paragraph{ExplicitStageDetector.}
Stage = value of an explicit metadata field. For BEAM, this is
\texttt{plan\_id} (the dataset provides plan indices per turn, with
each plan representing a thematic conversation segment). For
production use, this could be \texttt{agent\_topic},
\texttt{project\_directory}, or any other session-level label.
\paragraph{TemporalStageDetector.}
Stage = contiguous block of memories where inter-memory temporal gaps
are below a threshold (default: 4 hours). When a gap exceeds the
threshold, a new stage begins. The 4-hour default matches a typical
work-session boundary.
\paragraph{CompositeStageDetector.}
Tries detectors in priority order, falling back from explicit to
temporal when labels are absent. This is the recommended production
configuration.
The pluggable design enables A/B testing of stage strategies without
modifying the assembly logic. For the experiments in this paper, we
use \texttt{ExplicitStageDetector} with \texttt{plan\_id} on BEAM
(where plan boundaries are ground truth) and note that production
deployment would use \texttt{CompositeStageDetector}.
\subsubsection{Phase 1: Own-Stage Retrieval with Submodular Coverage}
\label{sec:phase1}
Phase~1 retrieves from the current stage's memories. The procedure is:
\begin{enumerate}
\item \textbf{Candidate generation.} Query the WRRF pipeline with
a stage filter and 3$\times$ oversampling: retrieve
$\text{max\_chunks} \times 3$ candidates from the current stage.
\item \textbf{Submodular selection.} From the oversample, select
the final set via the MMR-submodular objective:
\begin{equation}
S^* = \operatorname*{arg\,max}_{|S| \leq k} \sum_{c \in S} \left[\text{score}(c) - \lambda \cdot \max_{c' \in S \setminus \{c\}} \text{sim}(c, c')\right]
\label{eq:mmr}
\end{equation}
where $\text{score}(c)$ is the WRRF score,
$\text{sim}(c, c')$ is cosine similarity over 384D embeddings,
and $\lambda = 0.5$ balances relevance against diversity.
\item \textbf{Greedy algorithm.} Iterate up to
$\text{max\_chunks}$ times. At each step, select the candidate
not yet in $S$ that maximizes the marginal gain.
\end{enumerate}
The greedy submodular selection procedure is given in
Algorithm~\ref{alg:submodular}.
\begin{algorithm}[t]
\caption{Submodular coverage selection (Phase~1)}
\label{alg:submodular}
\begin{algorithmic}[1]
\REQUIRE candidates, max\_chunks, $\lambda$
\STATE $S \leftarrow \emptyset$, $E \leftarrow \emptyset$ \COMMENT{selected set, selected embeddings}
\FOR{$i = 1$ \TO max\_chunks}
\STATE best\_idx $\leftarrow -1$, best\_gain $\leftarrow -\infty$
\FOR{$j = 0$ \TO $|\text{candidates}| - 1$}
\IF{$j \in S$}
\STATE \textbf{continue}
\ENDIF
\IF{$E = \emptyset$ \OR embedding[$j$] is null}
\STATE penalty $\leftarrow 0$
\ELSE
\STATE penalty $\leftarrow \max_{e \in E} \text{cosine}(\text{embedding}[j], e)$
\ENDIF
\STATE gain $\leftarrow \text{score}[j] - \lambda \cdot \text{penalty}$
\IF{gain $>$ best\_gain}
\STATE best\_gain $\leftarrow$ gain
\STATE best\_idx $\leftarrow j$
\ENDIF
\ENDFOR
\IF{best\_idx $< 0$}
\STATE \textbf{break}
\ENDIF
\STATE $S \leftarrow S \cup \{\text{best\_idx}\}$
\IF{embedding[best\_idx] is not null}
\STATE $E \leftarrow E \cup \{\text{embedding}[\text{best\_idx}]\}$
\ENDIF
\ENDFOR
\RETURN $\{\text{candidates}[i] : i \in \text{sorted}(S)\}$
\end{algorithmic}
\end{algorithm}
\paragraph{Approximation guarantee.}
By \citet{Krause2008}, the greedy algorithm achieves
$f(S_k) \geq (1 - 1/e) \cdot f(S_k^*) \approx 0.63 \cdot f(S_k^*)$
for any monotone submodular function under cardinality constraints.
The MMR objective with $\lambda < 1$ exhibits diminishing returns:
adding a candidate $c$ to a larger selected set $A$ incurs a
penalty $\lambda \cdot \max_{s \in A} \mathrm{sim}(c, s)$ that
can only increase as $|A|$ grows, so the marginal gain
$\Delta(c|A) = \mathrm{score}(c) - \lambda \cdot \max_{s \in A}
\mathrm{sim}(c, s)$ decreases monotonically with $|A|$. This
is the definition of submodularity, and the $(1 - 1/e)$
approximation guarantee of \citet{Krause2008} therefore applies.
\paragraph{Why submodular selection matters at scale.}
At BEAM-10M, the top-5 candidates from WRRF within a single stage
frequently overlap in content. A specific failure mode we observed:
three of five selected memories described the same debugging session
from slightly different timestamps. Submodular selection penalizes
this redundancy, forcing the selected set to cover more aspects of
the queried topic.
\paragraph{Decoupling selection from budget.}
A deliberate design choice: selection is always by count
(\texttt{max\_chunks}), never by token budget. This ensures retrieval
ranking metrics (MRR, R@$k$) are well-defined regardless of individual
memory sizes. Token budgeting is enforced at assembly time by the
ContextDecomposer, which may truncate individual chunks but never
reduces the count of selected items. The two concerns---``which
memories matter?'' and ``how many tokens can we spend?''---are
independent and should not contaminate each other.
This decoupling was itself a bug fix. The initial implementation
enforced token budgets at selection time, causing the assembler to
select fewer than \texttt{max\_chunks} items when individual memories
were long. This degraded retrieval recall (fewer items means fewer
chances for a hit) without improving prompt quality (the
ContextDecomposer would have condensed the long items anyway).
\subsubsection{Phase 2: Cross-Stage Retrieval via Personalized PageRank}
\label{sec:phase2}
Phase~2 addresses the question: ``What information from \emph{other}
stages is relevant to the current query?''
Flat dense retrieval over the entire corpus is the default answer,
but at 7,500 memories it degenerates (Section~\ref{sec:scaling-failure}).
Instead, we follow the HippoRAG approach \citep{Gutierrez2024} and
use entity co-occurrence as a structural bridge between stages.
\paragraph{Step 1: Entity extraction from Phase~1.}
From the Phase~1 selected memories, extract the set of entity IDs.
Each memory in Cortex is annotated with its contained entities at
ingest time (stored in the \texttt{entity\_memory} junction table).
The seed entity set is built by aggregating entity IDs across all
Phase~1 results, with each entity weighted by how many Phase~1
memories contain it:
\begin{equation}
s(e) = |\{m \in S_1 : e \in \text{entities}(m)\}|
\label{eq:seed}
\end{equation}
where $S_1$ is the Phase~1 selected set.
\paragraph{Step 2: Graph construction.}
Build an adjacency structure over Cortex's entity and relationship
tables. Each entity is a node; each relationship adds a bidirectional
weighted edge with strength proportional to the co-occurrence strength
stored in the database. Edge weights are normalized to transition
probabilities within the PPR computation.
\paragraph{Step 3: Personalized PageRank.}
Compute PPR scores via power iteration with restart probability
$\alpha = 0.15$ \citep{Brin1998}: