Skip to content

Commit 7b77fd4

Browse files
committed
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter and phase attribution with zero production call sites; this one migrates the memory that matters, proves where it goes, removes the waste the proof exposed, and makes the memory budget a promise: over budget, extraction results go to disk and the run completes at the floor instead of aborting. Every number below is from a real index on this machine (M5 Pro, 48 GB) with CBM_MEM_PHASES=1 and the release build. WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source) The graph buffer, blamed for the peak since the 2026-07 research, is 3% of the memory: 0.55 GB peak. The per-file extraction results are the memory: 14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest is temporaries -- cbm_node_text copies at 496 call sites, per-node QN strings, abandoned generations of every growable array -- that an arena cannot free and that the result kept alive until resolve ended, because the result owned the arena. The arenas were charged 28.8 GB of capacity for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731 results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because the C walk allocated its working set in the result arena and stored arena pointers into the shared registry. After that, 7 GB of the kernel's post-resolve memory had no class at all. WHAT CHANGED Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table and dyn_array join the table. Two cross-boundary frees of buffer-owned strings (pass_importance, pass_complexity) go through a new setter, cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916. Result compaction (internal/cbm/result_compact.c). At the end of a file's extraction, everything reachable from its result is measured, copied into one exact-size arena (strings interned by content within the file) and the working arena is handed back. Later appends restart growth at the default block (cbm_arena_init_exact, arena grow_size). The stale duplicate internal/cbm/arena.h, which shared the foundation header's include guard and won or lost by include order, is a shim now. Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots (256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40. The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One lifecycle for every language's cross-file resolve: a scratch arena, a per-file overlay registry chained to the sealed shared base, the walk against the overlay, resolved calls appended into the result, scratch destroyed. type_registry gains chain-aware iterators (every yielded index belongs to it->reg) and copy-on-write refinement (cbm_registry_func/type_for_update), and the C, C#, Go and Python walks use them. This is what an ASan heap-use-after-free in the lsp_resolution_probe suite demanded: the C walk stored per-call arena objects into the shared registry, and a kernel run with a naive scratch arena lost 1,045 edges to the resulting garbage reads. It also ends unsynchronized writes into the shared registry from parallel resolve workers. The 7 GB with no class, found and cleaned. gbuf_index tables and keys were 2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five secondary indexes they never query (cbm_gbuf_new_worker builds none), and 18M per-key index arrays started at 8 slots (cbm_da_push_min starts them at 2); edge property JSON is 81% "{}" (interned). The semantic pass allocated a 512-token stride per function up front (packed tokens with offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps one reusable working arena between files (cbm_work_arena_take/give: rewind, not free) and every phase mark returns freed pages to the OS. Budget metric. The over-budget check compares the charged footprint -- max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because pages mimalloc has purged stay resident until the OS wants them, and phys_footprint under-reports after MADV_FREE_REUSABLE cycles. Spill / admission control (internal/cbm/result_spill.c, the extract gate in pass_parallel.c). At 15/16 of the budget (or over it, or with CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate sees the crossing per file pull and the workers' in-flight files carry the charge past the line before the first sweep lands: every compacted result is parked on disk as header + block in one append-only file per writer under <cache>/spill/, the results already cached are swept out by every worker, and registry build, def collection, surface rows, resolve and the infra-route passes load a result only for the moment they read it (relocated by base delta). The abort path stays closed while a sweep runs or a cached result remains; it fires only at the floor -- graph + registries + in-flight files. A result that owns sub-results stays in memory; a retained parse tree is dropped with the parked result and re-parsed on load. Every consumer of the result cache reads a slot through cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked slot is NULL in the array, and the passes that indexed it directly lost every __route__infra__ node and its INFRA_MAPS edges before the contract. The other things the transient loads found: a def label that was the result's own pointer (SIGBUS in the Go cross registry builder), two JVM helpers that passed their input through, a dedup table whose keys borrowed sr->value (the insert spun on freed memory), and a latch that was visible before the store was open (a peer found nothing to park and failed the run while its neighbour wrote 9 GB). The arena rewind reuse branch had defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it reuses only blocks that exist. The incremental and probe routes still index the cache array themselves and therefore never spill (ctx->spill_allowed stays false there; follow-up). The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The pass held every function's tokens, the corpus's per-document token ids and the vectors being stored at once: 4.9 GB on the kernel, on top of the floor, and nothing sized it to the headroom. Now, when the charged footprint plus that transient would cross the budget, phases 2-4 run per batch of functions: tokenize -> count into the corpus -> free; finalize once; tokenize again -> vectorize -> store -> free, with pages returned to the OS after every batch. Tokenization is deterministic, so both passes see the same tokens and the graph does not change; the run pays with a second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the batched-equals-unbatched test). The corpus frees its per-document token ids at the end of finalize (the co-occurrence pass is their only reader), and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak bytes and the charged footprint at every sub-phase. The semantic corpus and tokenizer allocate through the core (54 raw sites -> 0). This is not attribution only: cbm_sem_tokenize handed out libc strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in the production build -- since the pass moved onto the core. No libc-backed test build can see a cross-allocator free, so the core now refuses a block it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set, fatally: every proof run in this PR ran with it armed, and the Go and kernel indexes report zero foreign blocks. Accounting without contention. The core charged every allocation and free to shared per-class atomics; with 18 workers on 18 cores every block bounced the same three cache lines, and the bench against the shipped v0.10.8 showed it as CPU time rising faster than wall time for the same graph: Kotlin 38 -> 121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The deltas are thread-local now and reach the shared counters once per 256 KB or 512 blocks of change per thread, when a parallel-for worker ends, and whenever a reader looks (its own thread first). Phase marks read after the join and are exact; a class peak is a diagnostic and can be low by threads x 256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s (v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged. SQLite on a heap of its own. The same bench put the kernel's write step at 132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish statement. A stack sample named the cost: mimalloc's free-page search and its periodic heap collect, walking the graph's pages -- millions of them once the graph lives on the core -- for every statement-journal chunk SQLite allocates. SQLite now allocates from a mimalloc heap of its own per thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes), and the periodic collect is set to the maximum mimalloc allows since the explicit release at phase marks and spill sweeps is where memory goes back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s (v0.10.8: 296 s). Formatting without a process lock. On macOS every vsnprintf takes the process locale under an unfair lock (localeconv_l inside __vfprintf); eighteen workers formatting type names on the C# corpus collapsed into it -- the 10 MB JIT test files took 68 s each instead of under a second -- and cbm_arena_sprintf paid it twice per string (size pass, write pass). Each thread now formats with a C locale object of its own (vsnprintf_l), and the common short string is formatted once into a stack buffer. Per-file budgets that cover the whole file. The 5 s per-file budget covered the parse only; the LSP walks after it had none. C# JIT stress files (a 23 MB single expression among them) parsed inside the budget -- the old wall-clock parse timeout used to drop them, which hid the rest -- and then held a worker for 346 s each in the usage walk (tree-sitter's ts_node_parent descends from the root: quadratic on a deep tree), and the cross-file resolve ran on the same trees. Three rules now, one site for every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a parse that used more than half the budget disqualifies the file from the LSP walks (lsp_skipped); the unified walk checks its thread CPU time every 1024 nodes against six budgets and stops there, keeping what it found (walk_truncated, implies lsp_skipped); a file whose parse plus walk spent the budget is skipped by the LSP walks as well. Each decision is logged with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction 355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the 15 MB System.Runtime.Intrinsics reference file walks to the end and, with four generic-nesting JIT regression tests, only skips the LSP walk under the parse-plus-walk rule. The crash behind the C# bench. dotnet/runtime is 42,555 files, and its resolve phase died with SIGBUS on every run of this branch. A libc-backed ASan build of the server named it: heap-use-after-free in c_adl_resolve, reading a type name that another worker had allocated in its per-file scratch arena and freed at the end of its file. The C++ class walk refines a method's return type when the declaration is more specific than the pre-registered one (NAMED -> pointer/reference/template), and it did that by casting the chained lookup result to non-const and writing a scratch-arena signature into it -- into the sealed shared base when the entry lived there. The overlay contract had exactly one bypass, and it was this cast; every lookup returns const, so a grep for the cast is the audit. The refinement now goes through cbm_registry_func_for_update: copy-on-write into the overlay, the base untouched. Test: clsp_method_return_refinement_is_copy_on_write (a sealed base with a NAMED method return, an in-class pointer declaration; the base signature pointer is unchanged and the overlay copy carries the refinement plus a marker only a copy has) -- RED on the cast, GREEN on the accessor. The fixed ASan build then indexed the whole corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker exit 0, no report. Production build backs the core with mimalloc explicitly (never the global override, which macOS's two-level namespace forbids); the test build keeps libc so the sanitizers see every block -- the same split the tree-sitter and SQLite bindings already use. PROOF Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856 MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run jitter (gRPC Route nodes: two identical runs differ by 21). Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875 surface rows, nodes/edges inside the Route jitter. Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep took the charge to 768 MB, the run completed at 1.99 GB peak RSS. Kernel DEFAULT budget (24,576 MB): aborted before, completes now without spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1 GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges inside the +-400 run-to-run band the kernel shows between identical runs. Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3 GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6 GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and outside what spill can move). Nodes identical; edges 15,773,598. Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes, every phase mark under 15,000 MB by every metric, but the process RSS high-water mark reached 16.9 GB inside the semantic pass (class peak 4.9 GB against 0.8 GB live). Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint stays under 15,000 MB through the whole semantic pass (14,018 -> 14,381 MB across its sub-phases, 6 batches of 125,644 functions); every phase mark is under budget by every metric (max footprint 12.6 GB, max commit 11.5 GB); the process RSS high-water mark is 15.65 GB and is set in extraction, at the moment the gate latches (15,002 MB charged) and before the first sweep lands -- a 4% overshoot of in-flight work, no longer the semantic pass. Nodes identical, edges 15,772,913, zero foreign blocks. Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode enters at 14,062 MB (near_budget); the charged footprint never exceeds 11.8 GB at any mark; the process RSS high-water mark is 15.1 GB in extraction and 15.3 GB during the final write-out -- 1-2% over, the in-flight window and the dump transient. Nodes identical, edges 15,772,543, zero foreign blocks. Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction peak 998 MB RSS, graph identical to the baseline. BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14) corpus wall s peak RSS GB nodes / edges / CALLS perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3 php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8% rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369 c 5.9 -> 5.7 0.16 -> 0.19 identical kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1% django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4% go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2% java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6% csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6% typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130 kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1% No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 -> 1,403 s). Every graph delta against v0.10.8 is main between the release and the branch base (339b3f4), not this PR: a build of the base commit run through the same driver reproduces the final graphs exactly on django and kotlin, within the pre-existing gRPC Route jitter on go (30 nodes, 27 edges, CALLS identical), and on C# with the same node count. This PR's own C# difference against the base is 5 CALLS (the five truncated JIT stress files) plus one ijwhost swap and 563 edges of 5.79 M; the base takes 448 s / 29.7 GB for that corpus. Diagnostics that made this measurable stay in: the extract.arenas and extract.census lines (behind CBM_MEM_PHASES=1), the CBM_MEM_RELEASE=1 release-to-OS probe, and peak_charged_mb on every mem.phase line -- the high-water mark of the budget metric itself (cbm_mem_peak_charged), next to the RSS peak that counts pages already purged but not yet reclaimed. Tests: extract_compact_* and extract_spill_round_trip_keeps_every_field (extraction), parallel_spill_mode_builds_the_same_graph (parallel: every file parked, node count and every edge type equal to the in-memory run), pipeline_semantic_batched_matches_unbatched (pipeline: one repo indexed unbatched and with CBM_SEM_BATCH=5 -- byte-identical node vectors, token vectors and SEMANTICALLY_RELATED edges), extract_lsp_skipped_when_parse_used_ its_budget_share and extract_walk_truncated_at_its_cpu_budget (extraction: the seams mark a file / stop the walk, the LSP walk and the dispatcher skip it, the defs stay), registry_overlay_chain_iterates_ and_copies_on_write (c_lsp), arena exact-init/growth, mem charged/footprint/ pressure. Not in this increment: run-to-run edge jitter (+-400 on the kernel, Route class on Go) predates this work and has its own item; the incremental route spilling; a tighter result encoding (def ids instead of QN strings in resolved calls) is the next lever on the 9 GB the kernel's results still take. Signed-off-by: Martin Vogel <[email protected]>
1 parent b5f10c2 commit 7b77fd4

54 files changed

Lines changed: 4486 additions & 846 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile.cbm‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -276,6 +276,8 @@ EXTRACTION_SRCS = \
276276
$(CBM_DIR)/extract_k8s.c \
277277
$(CBM_DIR)/extract_dbt.c \
278278
$(CBM_DIR)/helpers.c \
279+
$(CBM_DIR)/result_compact.c \
280+
$(CBM_DIR)/result_spill.c \
279281
$(CBM_DIR)/lang_specs.c \
280282
$(CBM_DIR)/macro_table.c \
281283
$(CBM_DIR)/iris_export_xml.c \

‎internal/cbm/arena.h‎

Lines changed: 7 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,7 @@
1-
#ifndef CBM_ARENA_H
2-
#define CBM_ARENA_H
3-
4-
#include <stddef.h>
5-
6-
// CBMArena is a simple bump allocator that allocates from fixed-size blocks.
7-
// All memory is freed at once via cbm_arena_destroy(). Individual frees are not
8-
// supported — this is by design for per-file extraction where all data has the
9-
// same lifetime.
10-
#define CBM_ARENA_MAX_BLOCKS 256
11-
#define CBM_ARENA_DEFAULT_BLOCK_SIZE (64 * 1024) // 64KB initial
12-
13-
typedef struct {
14-
char *blocks[CBM_ARENA_MAX_BLOCKS];
15-
size_t block_sizes[CBM_ARENA_MAX_BLOCKS]; // per-block sizes (for stats)
16-
int nblocks;
17-
size_t block_size;
18-
size_t used; // bytes used in current block
19-
size_t total_alloc; // cumulative bytes allocated (for stats)
20-
} CBMArena;
21-
22-
// Initialize an arena with the default block size.
23-
void cbm_arena_init(CBMArena *a);
24-
25-
// Allocate n bytes from the arena. Returns NULL on OOM or block exhaustion.
26-
// All returned pointers are 8-byte aligned.
27-
void *cbm_arena_alloc(CBMArena *a, size_t n);
28-
29-
// Duplicate a string into arena memory. Returns arena-owned copy.
30-
char *cbm_arena_strdup(CBMArena *a, const char *s);
31-
32-
// Duplicate a string of known length into arena memory. NUL-terminates.
33-
char *cbm_arena_strndup(CBMArena *a, const char *s, size_t len);
34-
35-
// sprintf into arena memory. Returns arena-owned string.
36-
char *cbm_arena_sprintf(CBMArena *a, const char *fmt, ...) __attribute__((format(printf, 2, 3)));
37-
38-
// Free all blocks. Arena is invalid after this call.
39-
void cbm_arena_destroy(CBMArena *a);
40-
41-
#endif // CBM_ARENA_H
1+
/* The arena lives in src/foundation/arena.h. This file used to be a stale
2+
* copy with the same include guard and a shorter API; whichever header a
3+
* translation unit included first won, silently. One definition now. */
4+
#ifndef CBM_INTERNAL_ARENA_SHIM_H
5+
#define CBM_INTERNAL_ARENA_SHIM_H
6+
#include "../../src/foundation/arena.h" /* relative: the lsp_all unit has no -Isrc */
7+
#endif /* CBM_INTERNAL_ARENA_SHIM_H */

‎internal/cbm/cbm.c‎

Lines changed: 166 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
/* Full declaration set for the same CBMArena, and it must precede cbm.h:
22
* internal/cbm/arena.h declares a subset and the two share the CBM_ARENA_H
33
* guard, so whichever is included first is the one this file sees. */
4-
#include "foundation/arena.h" // cbm_arena_init_sized
4+
#include "foundation/arena.h" // cbm_arena_init_sized
5+
#include "foundation/mem_core.h" // class accounting for the bound allocators
6+
#include "foundation/log.h" // cbm_log_warn -- extract.lsp.skipped
57
#include "cbm.h"
68
#include "arena.h" // CBMArena, cbm_arena_init/alloc/strdup/destroy
79
#include "helpers.h"
@@ -222,6 +224,13 @@ static const char *cbm_string_read(void *payload, uint32_t byte, TSPoint point,
222224
* edge). A generous WALL ceiling stays as a backstop so a genuinely
223225
* stuck/spinning parse still terminates in bounded time. */
224226
#define CBM_PARSE_WALL_CEILING_FACTOR 12ULL /* ~60 s ceiling for the 5 s CPU budget */
227+
/* A parse that used more than 1/N of the per-file budget disqualifies the file
228+
* from the unbudgeted LSP walks (see cbm_extract_file_ex). */
229+
#define CBM_LSP_BUDGET_SHARE_DIV 2ULL
230+
/* The unified walk may spend this many parse budgets of thread CPU time: wide
231+
* enough for a 7,873-definition reference file (~10 s), tight enough to stop the
232+
* generated JIT tests (65-350 s). */
233+
#define CBM_WALK_BUDGET_FACTOR 6ULL
225234

226235
typedef struct {
227236
uint64_t cpu_deadline_ns; // trip once this thread's CPU time passes it
@@ -303,16 +312,43 @@ static TSParser *get_thread_parser(const TSLanguage *ts_lang, CBMLanguage lang)
303312
* hook here the biggest per-request allocations in the process — SQLite's page
304313
* cache and its query working set — are invisible to the attribution profile
305314
* (#581). */
315+
/* SQLite allocates from a mimalloc heap of its own, one per thread. Its page
316+
* queues then hold SQLite blocks only. Sharing the thread's default heap with
317+
* the graph -- 40M+ blocks, millions of pages once the graph moved onto the
318+
* core -- made every statement-journal chunk of the coverage publish step pay
319+
* a walk over the graph's pages: 132 s on the kernel where v0.10.8, whose
320+
* graph lived outside mimalloc, took 9.7 s (sampled 2026-09-14). mi_free
321+
* works across heaps, so xFree and cross-thread frees are unchanged; a
322+
* thread's heap is released with the thread. */
323+
static _Thread_local mi_heap_t *tl_sqlite_heap;
324+
325+
static mi_heap_t *sqlite_heap(void) {
326+
if (!tl_sqlite_heap) {
327+
tl_sqlite_heap = mi_heap_new();
328+
}
329+
return tl_sqlite_heap ? tl_sqlite_heap : mi_heap_main();
330+
}
331+
306332
static void *cbm_sqlite_malloc(int n) {
307-
void *block = mi_malloc((size_t)n);
333+
void *block = mi_heap_malloc(sqlite_heap(), (size_t)n);
334+
if (block) {
335+
cbm_mem_class_add_external(CBM_MEM_CLASS_STORE, mi_usable_size(block));
336+
}
308337
return block;
309338
}
310339
static void cbm_sqlite_free(void *p) {
340+
if (p) {
341+
cbm_mem_class_remove_external(CBM_MEM_CLASS_STORE, mi_usable_size(p));
342+
}
311343
mi_free(p);
312344
}
313345
static void *cbm_sqlite_realloc(void *p, int n) {
314-
if (p) {}
315-
void *grown = mi_realloc(p, (size_t)n);
346+
size_t old_size = p ? mi_usable_size(p) : 0;
347+
void *grown = mi_heap_realloc(sqlite_heap(), p, (size_t)n);
348+
if (grown) {
349+
cbm_mem_class_remove_external(CBM_MEM_CLASS_STORE, old_size);
350+
cbm_mem_class_add_external(CBM_MEM_CLASS_STORE, mi_usable_size(grown));
351+
}
316352
return grown;
317353
}
318354
static int cbm_sqlite_size(void *p) {
@@ -325,18 +361,31 @@ static int cbm_sqlite_roundup(int n) {
325361
* through these, and they too skip the interposer. */
326362
static void *cbm_ts_malloc(size_t n) {
327363
void *block = mi_malloc(n);
364+
if (block) {
365+
cbm_mem_class_add_external(CBM_MEM_CLASS_TS_TREE, mi_usable_size(block));
366+
}
328367
return block;
329368
}
330369
static void *cbm_ts_calloc(size_t count, size_t size) {
331370
void *block = mi_calloc(count, size);
371+
if (block) {
372+
cbm_mem_class_add_external(CBM_MEM_CLASS_TS_TREE, mi_usable_size(block));
373+
}
332374
return block;
333375
}
334376
static void *cbm_ts_realloc(void *p, size_t n) {
335-
if (p) {}
377+
size_t old_size = p ? mi_usable_size(p) : 0;
336378
void *grown = mi_realloc(p, n);
379+
if (grown) {
380+
cbm_mem_class_remove_external(CBM_MEM_CLASS_TS_TREE, old_size);
381+
cbm_mem_class_add_external(CBM_MEM_CLASS_TS_TREE, mi_usable_size(grown));
382+
}
337383
return grown;
338384
}
339385
static void cbm_ts_free(void *p) {
386+
if (p) {
387+
cbm_mem_class_remove_external(CBM_MEM_CLASS_TS_TREE, mi_usable_size(p));
388+
}
340389
mi_free(p);
341390
}
342391

@@ -1575,13 +1624,12 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
15751624
const CBMReturnTypeTable *return_type_table,
15761625
CBMArena *scratch) {
15771626
// Allocate result on heap (arena inside for all string data)
1578-
enum { SINGLE = 1 };
1579-
CBMFileResult *result = (CBMFileResult *)calloc(SINGLE, sizeof(CBMFileResult));
1627+
CBMFileResult *result = cbm_result_alloc();
15801628
if (!result) {
15811629
return NULL;
15821630
}
15831631

1584-
cbm_arena_init(&result->arena);
1632+
cbm_work_arena_take(&result->arena);
15851633
CBMArena *a = &result->arena;
15861634

15871635
/* Crash-quarantine hard guard (Stage 3c): a file the supervisor pinned as a
@@ -1642,6 +1690,7 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
16421690
ts_parser_reset(parser);
16431691

16441692
uint64_t t0 = now_ns();
1693+
uint64_t cpu_start_ns = cbm_thread_cpu_time_ns();
16451694

16461695
// Build string input + timeout options for parse_with_options
16471696
CBMStringInput str_input = {source, (uint32_t)source_len};
@@ -1654,8 +1703,9 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
16541703

16551704
TSParseOptions opts = {0};
16561705
CBMParseBudget budget = {0}; // cppcheck-suppress unreadVariable
1706+
uint64_t budget_ns = 0;
16571707
if (timeout_micros > 0) {
1658-
uint64_t budget_ns = (uint64_t)timeout_micros * USEC_TO_NSEC;
1708+
budget_ns = (uint64_t)timeout_micros * USEC_TO_NSEC;
16591709
// Descheduling burns wall time but not CPU: gate on this thread's CPU
16601710
// time so a starved-but-parseable file is not abandoned, with a generous
16611711
// wall ceiling as a backstop against a genuinely spinning/stuck parse.
@@ -1688,6 +1738,33 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
16881738

16891739
TSNode root = ts_tree_root_node(tree);
16901740

1741+
/* Parse-budget share. A file whose parse alone consumed more than
1742+
* 1/CBM_LSP_BUDGET_SHARE_DIV of its budget is too large for the per-file
1743+
* LSP walk that follows: the walk is superlinear in expression size and
1744+
* has no budget of its own (C#, a 23 MB single-expression JIT test: 354 s
1745+
* in the walk, then a crash in the cross-file resolve on the same tree,
1746+
* 2026-09-14 -- the parse used to time out at 5 s and hide both). The
1747+
* unified extractor's defs stay; the LSP refinement here and the
1748+
* cross-file resolve (cbm_pxc_dispatch_file) skip the file, logged. The
1749+
* budget is the same for every parser, so the rule is too. */
1750+
bool lsp_skipped = timeout_micros > 0 && (t1 - t0) * CBM_LSP_BUDGET_SHARE_DIV > budget_ns;
1751+
#ifdef CBM_ENABLE_TEST_SEAMS
1752+
{
1753+
const char *skip_on = getenv("CBM_TEST_LSP_SKIP_ON");
1754+
if (skip_on && skip_on[0] && rel_path && strstr(rel_path, skip_on)) {
1755+
lsp_skipped = true; /* the test names the file; no real timing involved */
1756+
}
1757+
}
1758+
#endif
1759+
if (lsp_skipped) {
1760+
char parse_ms[CBM_SZ_32];
1761+
snprintf(parse_ms, sizeof(parse_ms), "%llu",
1762+
(unsigned long long)((t1 - t0) / CBM_NSEC_PER_MSEC));
1763+
cbm_log_warn("extract.lsp.skipped", "reason", "parse_budget", "parse_ms", parse_ms, "path",
1764+
rel_path ? rel_path : "");
1765+
result->lsp_skipped = true;
1766+
}
1767+
16911768
// Compute module QN. Java/Go derive the module from the CONTAINING
16921769
// DIRECTORY (package semantics) rather than baking the filename stem in,
16931770
// so def QNs, the LSP caller_qn, and the textual calls-enclosing QN all
@@ -1710,13 +1787,30 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
17101787
.root = root,
17111788
.macro_table = macro_table,
17121789
.return_type_table = return_type_table,
1790+
.walk_deadline_cpu_ns =
1791+
timeout_micros > 0 ? cbm_thread_cpu_time_ns() + budget_ns * CBM_WALK_BUDGET_FACTOR : 0,
17131792
};
17141793

17151794
// Run extractors: defs + imports use separate walks (unique recursion patterns),
17161795
// then a single unified cursor walk handles the remaining 7 extractors.
17171796
cbm_extract_definitions(&ctx);
17181797
cbm_extract_imports(&ctx);
17191798
cbm_extract_unified(&ctx);
1799+
if (ctx.walk_budget_exhausted) {
1800+
result->walk_truncated = true;
1801+
result->lsp_skipped = true;
1802+
cbm_log_warn("extract.walk.truncated", "reason", "cpu_budget", "path",
1803+
rel_path ? rel_path : "");
1804+
}
1805+
/* A file that spent the budget on parse plus walk is too heavy for the
1806+
* unbudgeted LSP walks as well (the C# JIT test files: 65-73 s each in
1807+
* the per-file walk after a parse under the share rule). */
1808+
if (!result->lsp_skipped && timeout_micros > 0 &&
1809+
cbm_thread_cpu_time_ns() - cpu_start_ns > budget_ns) {
1810+
result->lsp_skipped = true;
1811+
cbm_log_warn("extract.lsp.skipped", "reason", "file_budget", "path",
1812+
rel_path ? rel_path : "");
1813+
}
17201814

17211815
// Channel detection (Socket.IO / EventEmitter) — JS/TS only.
17221816
cbm_extract_channels(&ctx);
@@ -1736,7 +1830,7 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
17361830
// LSP type-aware call/usage resolution (per-file). Runs in every mode;
17371831
// refines the tree-sitter + textual-resolution graph with type info.
17381832
uint64_t lsp_start = now_ns();
1739-
{
1833+
if (!result->lsp_skipped) {
17401834
if (language == CBM_LANG_GO) {
17411835
cbm_run_go_lsp(a, result, source, source_len, root);
17421836
}
@@ -1775,15 +1869,15 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
17751869
if (language == CBM_LANG_CSHARP) {
17761870
cbm_run_cs_lsp(a, result, source, source_len, root);
17771871
}
1778-
}
1779-
if (language == CBM_LANG_JAVA) {
1780-
cbm_run_java_lsp(a, result, source, source_len, root);
1781-
}
1782-
if (language == CBM_LANG_KOTLIN) {
1783-
cbm_run_kotlin_lsp(a, result, source, source_len, root);
1784-
}
1785-
if (language == CBM_LANG_RUST) {
1786-
cbm_run_rust_lsp(a, result, source, source_len, root);
1872+
if (language == CBM_LANG_JAVA) {
1873+
cbm_run_java_lsp(a, result, source, source_len, root);
1874+
}
1875+
if (language == CBM_LANG_KOTLIN) {
1876+
cbm_run_kotlin_lsp(a, result, source, source_len, root);
1877+
}
1878+
if (language == CBM_LANG_RUST) {
1879+
cbm_run_rust_lsp(a, result, source, source_len, root);
1880+
}
17871881
}
17881882
atomic_fetch_add(&total_lsp_ns, now_ns() - lsp_start);
17891883

@@ -2156,6 +2250,40 @@ static CBMFileResult *extract_file_ex_body(const char *source, int source_len, C
21562250
return result;
21572251
}
21582252

2253+
/* ── Per-worker working arena (see cbm.h) ───────────────────────────── */
2254+
static CBM_TLS CBMArena tl_work_arena;
2255+
static CBM_TLS bool tl_work_arena_live = false;
2256+
2257+
void cbm_work_arena_take(CBMArena *into) {
2258+
if (tl_work_arena_live) {
2259+
*into = tl_work_arena;
2260+
tl_work_arena_live = false;
2261+
cbm_arena_rewind(into);
2262+
return;
2263+
}
2264+
cbm_arena_init(into);
2265+
}
2266+
2267+
void cbm_work_arena_release(void) {
2268+
if (tl_work_arena_live) {
2269+
cbm_arena_destroy(&tl_work_arena);
2270+
tl_work_arena_live = false;
2271+
}
2272+
}
2273+
2274+
void cbm_work_arena_give(CBMArena *from) {
2275+
if (!from || from->nblocks == 0) {
2276+
return;
2277+
}
2278+
if (tl_work_arena_live || cbm_arena_capacity(from) > (size_t)CBM_WORK_ARENA_KEEP_BYTES) {
2279+
cbm_arena_destroy(from);
2280+
return;
2281+
}
2282+
tl_work_arena = *from;
2283+
tl_work_arena_live = true;
2284+
memset(from, 0, sizeof(*from));
2285+
}
2286+
21592287
/* Public entry. Owns the traversal scratch arena for the whole of one file's
21602288
* extraction: created here, handed to the body as ctx->scratch, destroyed on
21612289
* the way out. The body has seven early returns, so bracketing it in a wrapper
@@ -2176,20 +2304,34 @@ CBMFileResult *cbm_extract_file_ex(const char *source, int source_len, CBMLangua
21762304
return result;
21772305
}
21782306

2179-
void cbm_free_result(CBMFileResult *result) {
2307+
CBMFileResult *cbm_result_alloc(void) {
2308+
/* The one raw allocation of a result: cbm_free_result releases it with
2309+
* the matching free. Extraction and the spill loader both come here. */
2310+
enum { SINGLE = 1 };
2311+
return (CBMFileResult *)calloc(SINGLE, sizeof(CBMFileResult));
2312+
}
2313+
2314+
void cbm_result_release_owned(CBMFileResult *result) {
21802315
if (!result) {
21812316
return;
21822317
}
2183-
if (result->cached_tree) {
2184-
ts_tree_delete(result->cached_tree);
2185-
result->cached_tree = NULL;
2186-
}
21872318
for (int i = 0; i < result->owned_result_count; i++) {
21882319
cbm_free_result(result->owned_results[i]);
21892320
}
21902321
free(result->owned_results);
21912322
result->owned_results = NULL;
21922323
result->owned_result_count = 0;
2324+
}
2325+
2326+
void cbm_free_result(CBMFileResult *result) {
2327+
if (!result) {
2328+
return;
2329+
}
2330+
if (result->cached_tree) {
2331+
ts_tree_delete(result->cached_tree);
2332+
result->cached_tree = NULL;
2333+
}
2334+
cbm_result_release_owned(result);
21932335
cbm_arena_destroy(&result->arena);
21942336
free(result);
21952337
}

0 commit comments

Comments
 (0)