Skip to content

perf(importance): warm recompute is dominated by a 1.13M-row UPDATE, not by scoring #1889

Open

Description

@DeusData

The staging-store importance recompute (#1883) costs +17.0 s on a kernel-scale warm reindex — measured, 48.5 s → 65.5 s at 8,529,900 nodes. That was accepted as the price of correct scores, but the cost looks avoidable without giving anything up.

The observation

The same scoring rule costs:

path cost
cold, in-memory pass 2.2 s
warm, SQL over the staging store ~20.6 s

Roughly 10×, for the same 8.5M-node graph. The in-memory path walks hash indexes already in RAM; the SQL path pays for two GROUP BYs over ~16M edges plus a 1.13M-row UPDATE.

So the dominant cost is write volume, not computation.

The hypothesis

A one-file incremental edit changes very few scores. If the UPDATE touched only rows whose score actually moved, most of the 20 s should disappear.

Crucially this preserves exact full-vs-incremental equivalence, which is the property that makes the two scoring paths trustworthy and is pinned by importance_full_and_incremental_converge_on_same_scores. That is what separates it from the rejected alternative of scoping the rescore to changed files — that one is provably wrong (editing c0.py's body changes helper's in-degree in unchanged hot.py: √13 vs √12), whereas this changes only which rows get written, not which get scored.

Not attempted

This is a hypothesis from a timing observation, not a measured result. Anyone picking it up should:

  1. Confirm the write is genuinely dominant — split the recompute's ~20.6 s into aggregate time vs UPDATE time before optimising the wrong half.
  2. Keep importance_full_and_incremental_converge_on_same_scores passing; if it goes red, the optimisation changed scores rather than writes.
  3. Re-measure the kernel warm A/B the same way: fixed corpus, one -O2 binary, serialized, route asserted closure_repair, node counts confirmed identical across legs.
  4. Watch for a WHERE clause that itself costs more than the writes it avoids — comparing 1.13M computed scores against stored values is not free.

Context

Nothing reads the score until #880, so today this is a cost paid ahead of the benefit. That makes the optimisation more attractive than the raw percentage suggests, but it is also why it did not block shipping #1883.

Activity

  1. added
    enhancementNew feature or request
    priority/normalStandard review queue; useful PR with ordinary maintainer urgency.
    stability/performanceServer crashes, OOM, hangs, high CPU/memory
    maintainer-notesInternal observations and improvement notes
    on Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmaintainer-notesInternal observations and improvement notespriority/normalStandard review queue; useful PR with ordinary maintainer urgency.stability/performanceServer crashes, OOM, hangs, high CPU/memory

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions