Repository navigation
Description
The staging-store importance recompute (#1883) costs +17.0 s on a kernel-scale warm reindex — measured, 48.5 s → 65.5 s at 8,529,900 nodes. That was accepted as the price of correct scores, but the cost looks avoidable without giving anything up.
The observation
The same scoring rule costs:
| path | cost |
|---|---|
| cold, in-memory pass | 2.2 s |
| warm, SQL over the staging store | ~20.6 s |
Roughly 10×, for the same 8.5M-node graph. The in-memory path walks hash indexes already in RAM; the SQL path pays for two GROUP BYs over ~16M edges plus a 1.13M-row UPDATE.
So the dominant cost is write volume, not computation.
The hypothesis
A one-file incremental edit changes very few scores. If the UPDATE touched only rows whose score actually moved, most of the 20 s should disappear.
Crucially this preserves exact full-vs-incremental equivalence, which is the property that makes the two scoring paths trustworthy and is pinned by importance_full_and_incremental_converge_on_same_scores. That is what separates it from the rejected alternative of scoping the rescore to changed files — that one is provably wrong (editing c0.py's body changes helper's in-degree in unchanged hot.py: √13 vs √12), whereas this changes only which rows get written, not which get scored.
Not attempted
This is a hypothesis from a timing observation, not a measured result. Anyone picking it up should:
- Confirm the write is genuinely dominant — split the recompute's ~20.6 s into aggregate time vs
UPDATEtime before optimising the wrong half. - Keep
importance_full_and_incremental_converge_on_same_scorespassing; if it goes red, the optimisation changed scores rather than writes. - Re-measure the kernel warm A/B the same way: fixed corpus, one
-O2binary, serialized, route assertedclosure_repair, node counts confirmed identical across legs. - Watch for a
WHEREclause that itself costs more than the writes it avoids — comparing 1.13M computed scores against stored values is not free.
Context
Nothing reads the score until #880, so today this is a cost paid ahead of the benefit. That makes the optimisation more attractive than the raw percentage suggests, but it is also why it did not block shipping #1883.
Activity