Repository navigation
fix(extract): definitions walk stack grows without a ceiling - #2373
Merged
Merged
Conversation
walk_defs keeps its pending frames in a growable stack that was bounded by an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged "extract.walk_defs_capped" once and then stopped pushing. Children are pushed last-to-first, so a file wider than the ceiling lost its FIRST definitions: a work cap decided graph content. Fix: no ceiling. The stack doubles on demand, heap frames now come from the memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and raw free, and the doubling is overflow-guarded (int capacity and size_t byte count). The only thing that can stop the walk is an allocation failure, and that is loud: logged at error level as "extract.walk_stack_alloc_failed" (walker=walk_defs, pending, path) -- the same event the complexity walkers use -- and the file's result is marked has_error with "definitions walk: stack allocation failed", which the definitions passes already report as a per-file extract error. The arena-backed path (ctx->scratch) gets the same failure handling. Traversal order is unchanged. CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not documented anywhere (docs, --help, config, tests), so nothing else changes. extract_defs.c loses its raw free() call, so the memory-core baseline for the file drops from 5 to 4. Test (extraction suite): walk_defs_wide_file_extracts_every_definition extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256 set, and asserts every wd0..wd999 definition is present. RED on main (extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED again with the production change reverted. The extraction (379), complexity (5), ast_profile (10) and semantic (35) suites pass. Real input: Linux kernel, index-only, fresh cache and runtime dirs, same machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges, SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and edge set (type, source QN, target QN) hash identically -- no kernel file approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix), within run-to-run noise. Signed-off-by: Martin Vogel <[email protected]>
This was referenced Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The definitions walker (
walk_defsininternal/cbm/extract_defs.c) had a hard ceiling on its explicit stack,wd_stack_max(), overridable only through the undocumentedCBM_WALK_DEFS_MAX. When a file needed more pending frames than the cap allowed, the extra subtrees were dropped with a singleextract.walk_defs_cappedwarning. Definitions in those subtrees vanished from the graph without any error. A heap-allocation failure was worse: it dropped every pending frame silently.A work cap must not decide graph content, so this removes the ceiling:
INT_MAX/2, and the byte count must fit insize_t.cbm_realloc/cbm_free, class EXTRACT). The arena path (ctx->scratch) is unchanged.extract.walk_stack_alloc_failed, the same event the complexity walkers use;result->has_erroris set, so the pipeline reports a per-file extract error and keeps the definitions found so far.wd_stack_max(), the capped warning and theCBM_WALK_DEFS_MAXknob are removed. The knob was never documented or tested.extract_defs.cdrops from 5 to 4 raw sites.Proof
Test:
walk_defs_wide_file_extracts_every_definition(extraction suite) uses a C file with 1000 top-level functions, with the old cap forced to 256. It asserts that all 1000 are extracted.main: RED.wd0is missing afterextract.walk_defs_capped limit=256.extract_defs.creverted tomain: RED again, identical failure.Existing suites: extraction, complexity, ast_profile and semantic are all green, with no changes to existing tests.
Linux kernel (
torvalds/linux, index-only, same machine, one run each):A hash over every node (label, QN, span) and every edge (type, source, target) is identical. No kernel file comes near the old cap, so no delta was expected: this is a no-regression check, not the effect proof. The effect proof is the RED/GREEN test above.
Lint: clang-format (Homebrew LLVM) is clean on the changed lines, and
lint-memory-core --strictpasses.