Repository navigation
Agent-memory vector workload studies: exact against graph search, memory profile, and rebuild behaviour #585
Description
Problem
I want to measure the vector index the way an agent uses it: one insert per turn, reads right after, defaults against capped graph-build threads, other rebuild thresholds, an explicit build_graph_now() at idle, with query p50 and p99 overall and during rebuilds, read-your-writes misses, and recall against exact on the live set. The exact-versus-graph extension, the memory profile cells, and the exact-scan micro study belong to it.
The bindings half (asyncio facade, read-your-writes test, vector bulk I/O) is ArcadeData/arcadedb-python#306. This issue holds the measurement half.
Priority: P2 (memory profile cells P4).
Acceptance
- The exact-versus-graph extension prep and release run, the units from n500 up are scored
- The memory profile cells (main768 and d1536) run and are scored
- The exact-scan micro study runs, and its native probe is retired if the native-image smoke already answers the question
- The agent-memory study is rerun on the release pair at the larger unit
Results and the scorer output go in this issue as comments. Numbers are generated by script from the result files, never typed.
Activity
tae898 commentedon Nov 12, 2026
Results of the extension study for this issue, native package only (the JVM package ran on its own, one of its units needs a top-up that is running now, and it gets its own comment). Package
arcadedb26.11.1, enginea3a5d12457, one x86_64 laptop, efficiency cores 6 and 7, one insert and four queries per turn. Automatic rebuilds are off in every arm, so this says nothing about the engine defaults.Gis the LSM_VECTOR graph with an explicit rebuild at idle,Xis the SQL cosine scan,Dis a graph over the first 1000 rows plus the engine's own exact delta scan for the rest, andZisGagain as an A/A control. The figure is query p99 as a median of per-round ratios with a bootstrap 95% interval fromab.py; above 1 means slower thanG.First a correction to my own scoring. My scorer expected the arm
Xat n100000, where the design runs onlyG,Z, andD, so it endedWEAK(n100000 X q_p99_ms: n = 0 < 3 clean pairs) however good the rows were. I amended it on 2026-11-12, before I re-scored. The rows are byte for byte the same, and everyab.pystatistic is identical to the old output. The amendment also makes an A/A control without an interval count as not shown, where before it passed silently. Old scorer:WEAK. Amended scorer on the same rows: stillWEAK, but now for one real reason, the n500 A/A control below.The A/A control contains 1.00 on p99 and p50 at all five units. The graph beats the SQL scan on p99 from 1000 vectors up, and it also beats the engine's delta scan at every size, thinly at n1000.
Dis cheaper in total wall time at every size because it pays no graph builds (loop wallDagainstGfrom 0.578 to 0.919). The delta scan is also faster than the SQL scan over the same rows (XagainstDon p99: 2.705 to 3.84, n1000 to n30000). For the query tail at these sizes the data do not favor serving the delta by an exact scan and keeping the graph for the settled bulk.n500 is left out of the table.
XagainstGreads 0.003 [0.003, 0.003] (8 pairs), but that is the engine and not the scan: below 1000 nodes it rebuilds the graph synchronously on the first query after every write, so everyGandZrun reports 500 rebuilds andGhas a p99 of 738.37 ms against 2.04 ms for the scan. The p50 tells the ordinary story, 0.51 ms for the graph and 1.06 ms for the scan. So this is about the small-graph policy and not about scan versus graph, and I will look at whether it deserves an upstream issue once the study is settled.The n500 A/A control had 6 clean pairs and
ab.pyneeds 7 for an interval, so I ran one top-up of n500 only on the same pin. With the Z against G pairs of both runs there are 12 clean pairs, and the p99 interval is 0.998 [0.997, 1.000] and the p50 interval is 1.002 [0.977, 1.006]. Both contain 1.00 under the rule I wrote before the top-up ran (the p99 interval ends at 1.000, asab.pyprints it), so n500 counts, with the reading above.If you would like another size or arm, tell me which and I will add it as new rows.
tae898 commentedon Nov 12, 2026
Results of the extension study for this issue, JVM package (the native package is in my comment above). Package
arcadedb-jvm26.11.1, enginea3a5d12457, the same laptop, cores, workload, and arms as there:Gis the LSM_VECTOR graph with an explicit rebuild at idle,Xis the SQL cosine scan,Dis a graph over the first 1000 rows plus the engine's own exact delta scan for the rest, andZisGagain as an A/A control. The figure is query p99 as a median of per-round ratios with a bootstrap 95% interval fromab.py; above 1 means slower thanG. The scorer is the amended one I described in the native comment.The A/A control contains 1.00 on p99 and p50 at n500 to n30000. The graph beats the SQL scan on p99 from 1000 vectors up (X against G from 5.47 to 11.273). The delta scan is slower than the graph on p99 from 3000 vectors up (D against G 1.373 at n3000, 4.419 at n30000), and at n1000 it reads 0.837 [0.727, 0.975], where the native package read 1.107 [1.103, 1.111]. The delta scan beats the SQL scan over the same rows on p99 (X against D from 2.495 to 6.603, n1000 to n30000). Loop wall time of D against G reads 0.725 at n1000, 0.748 at n3000, 1.288 at n10000, 1.307 at n30000.
n500 is left out of the table.
XagainstGreads 0.012 [0.012, 0.012] (12 pairs), but that is the engine and not the scan: below 1000 nodes it rebuilds the graph synchronously on the first query after every write, so everyGandZrun reports 500 rebuilds andGhas a p99 of 557.37 ms against 6.71 ms for the scan. The p50 tells the ordinary story, 0.54 ms for the graph and 0.72 ms for the scan. So this is about the small-graph policy and not about scan versus graph, and I will look at whether it deserves an upstream issue once the study is settled.n100000 carries no verdict. Its A/A control is biased:
ZagainstGreads 1.067 [1.009, 1.150] on p99 over 14 clean pairs (Zworse in 11 and better in 3, p = 0.0574), and 0.995 [0.989, 1.001] on p50. The first run had already flagged it, so I ran one top-up of six rounds of that unit on the same pin, with the rule written before it ran: a unit whose A/A control differs stays out and the bias is reported as a finding about the method. So I report none of theX,D, orGreadings for it. The same arm run twice differs by about 7 percent at the tail at 100,000 vectors on this package, so a difference of that size between two different arms there could not be told from the method's own floor. I do not know the cause; several rounds of the first run had foreign CPU on the laptop, and the noise rule dropped those arms as written.After posting I checked whether the bias follows the run order. The arms already run in a rotated order, so a pair of arms is in both orders. Looking at every round in which both
GandZfinished (a descriptive look, without the noise rule),Zis worse thanGon p99 in 10 of 11 rounds whereGran first (median ratio 1.06) and in 7 of 11 rounds whereZran first (median ratio 1.075). So it is not a position effect, and I do not know what it is.If you would like another size or arm, tell me which and I will add it as new rows.
Edited 2026-11-13: the paragraph about the run order is new, and the offer at the end of the first version to rerun n100000 with the arms in the other order is withdrawn, because the arms already run in both orders.
tae898 commentedon Nov 12, 2026
I edited my JVM comment above. I had offered to rerun n100000 with the arms in the other order, but the arms already run in both orders, and the bias is there in both (the edited text has the numbers). So the offer is withdrawn, and the cause of the n100000 A/A bias is still unknown.