fix(pipeline): sequential definitions pass caches every result with its parse tree and working arena #2416

Version

built from source, main @ 2278498 (v0.11.0-186)

Platform

Linux (x64)

Install channel

Built from source

Binary variant

standard

What happened, and what did you expect?

On the sequential route, which a repo takes with 50 files or fewer or with CBM_WORKERS=1 (src/pipeline/pipeline.c:2684), cbm_pipeline_pass_definitions caches each file's raw result (pass_definitions.c:870-871) with its tree-sitter tree and uncompacted working arena until the run ends. The parallel worker frees the tree (pass_parallel.c:1233) and compacts the result (pass_parallel.c:1290) before caching. So peak memory on the sequential route grows by about 1.7 MB per small Python file instead of about 0.2 MB, and a small repo of large files takes this route at any worker count.

The fix frees the tree and compacts the result before caching, as the parallel worker does, and releases the thread's working arena when the pass ends. The graph is byte-identical on 7 sequential-route corpora. The cost is that the cross-file LSP pass re-parses these files, as it already does on the parallel route (268-275 ms to 528 ms on the 480-file corpus).

Reproduction

To see it, index 40 large .py files with default settings (I used 17.9 MiB from torch, tensorflow, scipy, wx and others), or any Python tree with CBM_WORKERS=1, and watch peak memory.

Logs

Diagnostics trajectory (memory / performance / leak issues)

Not a leak: the extra memory is held until one index run ends, so the table gives each run's peak memory rather than a trajectory.

Project scale (if relevant)

No response

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.