Repository navigation
perf(py-lsp): shared Python cross-registry build is O(methods × types) #2413
Description
Version
built from source, main @ 2278498 (v0.11.0-186)
Platform
Linux (x64)
Install channel
Built from source
Binary variant
standard
What happened, and what did you expect?
cbm_py_build_cross_registry passes Python defs to py_register_lsp_defs one at a time with idx_arena == NULL (internal/cbm/lsp/py_lsp.c:5130-5136). That skips the mid-build finalize (py_lsp.c:5004-5010), so every Method's receiver probe (py_lsp.c:5049) is a linear strcmp scan over all types registered so far (type_registry.c:830-834). On a Python-heavy tree the build runs single-threaded for over a minute while the other workers wait. Incremental reindexes repeat it whenever the closure exceeds 8 files (src/pipeline/pipeline_incremental.c:1325). This is a different path from #1527, which is per-file resolution (py_lsp.c:5095).
I have a fix that backs only the existence probe with a cbm_ht set of type QNs, as the Go (959f4de) and Rust (10b3b79) fixes did. It keeps the per-def order, so the graph is byte-identical and peak RSS is unchanged.
Reproduction
To reproduce, index generated modules of 10 classes with 8 methods each, where each class inherits the previous one and calls into an imported class. My real-world case is the .py/.pyi files of a Python 3.14 install (26,233 files, 40,923 classes, 222,253 methods), with site-packages renamed because discovery skips it and mpmath removed because it hits a separate bug.
| Linux x86_64, 32 logical CPUs, GCC 16 | main | fixed |
|---|---|---|
Python builder (lsp_cross_prepare.builders), Python 3.14 tree, 3 runs |
67-75 s | 0.20-0.25 s |
| Same builder, generated 2k / 8k / 32k classes | 0.11 / 1.5 / 22 s | 8 / 28 / 135 ms |
| Whole index wall time, Python 3.14 tree | 175-182 s | 113 s |
Logs
Diagnostics trajectory (memory / performance / leak issues)
Not a memory problem: one CPU-bound, single-threaded phase of a cold index run. The fix leaves peak RSS unchanged.
Project scale (if relevant)
No response
Confirmations
- I searched existing issues and this is not a duplicate.
- My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.
Activity
DeusData commentedon Oct 19, 2026
Thank you for isolating the shared cross-registry builder from per-file resolution and providing scaling measurements. The proposed existence set and unchanged-order claim are useful review evidence. No verified fix status is available from this triage yet.
Merge origin/main into perf/py-cross-registry-index
DeusData commentedon Oct 25, 2026
Thank you for your patience. The shared Python builder’s scaling and proposed existence set remain maintainer review work. Sorry for the delay; I’ll follow up soon.
DeusData commentedon Oct 30, 2026
Fixed on main by #2420 (merge commit a33878c): the shared Python cross-registry build now checks type membership in constant time, so lsp_cross_prepare drops by 94 to 97 percent on django, the Python 3.14 standard library and both together, with a byte-identical graph. Thank you, @Fieldnote-Echo, for the report and the fix!