SymDB: bound Extractor#extract_all per-call peak working set#5883
Conversation
Two changes that together reduce Extractor#extract_all peak memory from 74 MB to 61 MB on a 2500-user-class workload (-17.6%, measured via benchmarks/symbol_database_extraction.rb from #5798), with per-iteration wall time within noise. Streaming yield (extractor.rb, component.rb) extract_all now accepts an optional block. When a block is given, FILE scopes are yielded one at a time and the per-file tree node is shifted out of the in-progress hash as it is yielded, so the caller can free each scope after consuming it without retaining the full Array<Scope>. The entries hash from Pass 1 is also dropped before Pass 3 begins, so its UnboundMethod references become eligible for collection at the next GC instead of staying alive for the whole Pass 3 conversion. Without a block, extract_all still returns the full Array<Scope> as before \u2014 preserves the test/benchmark API. Component.extract_and_upload uses the block form and accumulates the extracted_count + targetable_count inline rather than walking the full array after extraction. count_targetable_methods was split so the per-scope helper count_targetable_methods_in_scope is reusable in the streaming loop (the wrapper that takes the full Array is kept). Skip redundant visibility scan (extractor.rb) Module#instance_methods(false) already returns both public and protected methods. group_methods_by_file was iterating instance_methods + protected_instance_methods + private_instance_methods with a uniq! to drop the duplicated protected entries. Iterating only instance_methods + private_instance_methods covers all three visibilities without the redundant scan or the merged-array allocation.
Typing analysisNote: Ignored files are excluded from the next sections.
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 5315293 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-06-10 18:58:07 Comparing candidate commit c0771a8 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.
|
Refactor extract_all from a 3-pass collect/build/convert that retained every module's UnboundMethods until the final conversion, into a 2-pass index/build flow that holds at most one file's working set at a time. Pass 1 (new build_per_file_index): - Walks ObjectSpace.each_object(Module) once. - For each user-code module, records (file_path, module_ref, method_name_symbols) per file the module contributes to. - Stores Symbol method names plus Module references only. No UnboundMethod objects retained between passes. The UnboundMethods allocated to read source_location become garbage as the inner loop ends. Pass 2 (new build_file_scope): - For each file in the index, resolves UnboundMethods just-in-time for the modules that contribute to it and builds the FILE scope with full nested MODULE/CLASS hierarchy (same output as the old extract_all). - When extract_all is called with a block, the index is drained destructively (index.shift), so each per-file entry becomes collectable as soon as its FILE scope is yielded and consumed. Output shape unchanged. All existing extract_all specs pass (the one that stubbed collect_extractable_modules was updated to stub build_per_file_index instead). Measured memory effect (taskset -c 0,1, no_turbo=1, vs PR baseline + the earlier A+D commit): Classes no-block(KB) block-streaming(KB) 1500 20,308 ~6,000 (-85% vs original A baseline) 2500 41,592 ~10,700 (-85%) 3500 50,032 ~12,900 (-85%) 5000 71,060 ~16,400 (-83%) 8000 107,996 ~23,300 (-85%) The block-form (used by Component) now scales at ~2.3 MB per 1000 additional user classes, down from ~14 MB previously. The remaining growth is mostly the Ruby runtime's own per-class footprint (method tables, constant slots) that the extractor walks but does not allocate \u2014 it is also visible in baseline RSS growth across the same sizes. The legacy collect_extractable_modules / build_file_trees / convert_trees_to_scopes / group_methods_by_file methods are kept for tests that exercise the old shape; extract_all no longer uses them.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ae28f1eb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Refactors the SymDB extraction pipeline to reduce peak per-call memory usage by switching Extractor#extract_all to a per-file streaming approach (optional block form) and by avoiding allocations during method visibility scanning.
Changes:
- Added a block/streaming form for
Extractor#extract_all, yielding FILE scopes one at a time while preserving the array-returning legacy behavior when no block is given. - Replaced the prior multi-structure extraction flow with a per-file index + per-file scope build path, resolving
UnboundMethods just-in-time. - Updated
Component#extract_and_uploadand relevant specs to use the streaming/yield form and accumulate counts during streaming.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lib/datadog/symbol_database/extractor.rb | Implements per-file index + per-file scope build and adds streaming block form to bound working set. |
| lib/datadog/symbol_database/component.rb | Switches initial extraction to streaming consumption and computes counts inline without retaining the full scope array. |
| spec/datadog/symbol_database/extractor_spec.rb | Updates a stub to match the new extraction entry point (build_per_file_index). |
| spec/datadog/symbol_database/component_spec.rb | Updates expectations/stubs to match streaming extract_all behavior (yielding scopes). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Apply review fixes to the per-file streaming index work:
- Fix shadowing-outer-local-variable warning at extract_all (Ruby 2.5/2.6)
by switching the block-form drain to while(pair = shift) + pair[0]/pair[1]
indexing so the else-branch's |path, file_entries| block params have no
outer-scope binding to shadow.
- Update RBS signatures: extract_all now accepts an optional block and
returns nil in block form; add declarations for build_per_file_index,
collect_method_names_by_file, build_file_scope, convert_tree_to_scope,
count_targetable_methods_in_scope; remove the now-dead
count_targetable_methods RBS entry.
- Remove count_targetable_methods (dead after extract_and_upload moved to
inline per-scope accumulation).
- Add trailing commas in build_file_scope's root hash and
convert_tree_to_scope's Scope.new call.
- Rewrite the extract_all header comment to match the new two-pass design
(build_per_file_index → build_file_scope) instead of the old three-pass
one.
- Update convert_tree_to_scope's @param doc to note both callers
(build_file_trees and build_file_scope).
- Add block-form tests in extractor_spec.rb covering: yield-per-FILE,
return-nil contract, scope equivalence with the non-block form, and
top-level rescue behavior under block form.
Verified locally:
- bundle exec steep check: no errors
- bundle exec rspec spec/datadog/symbol_database/{extractor,component,integration}_spec.rb:
181 examples, 0 failures
- ruby -W2 require of extractor.rb: no warnings
…r_map on 2.5/2.6 Two fixes on the per-file streaming index path: - Port the resolves_to_same_module? guard from collect_extractable_modules (added in #5872) over to build_per_file_index. The new production entry point was missing this check, so the autoload-registered-after-remove_const and remove_const-detached extract_all specs regressed once master was merged. The check is the same one-liner between safe_mod_name and user_code_module? as in the legacy method. - Replace bare method_names.filter_map at build_file_scope with Core::Utils::EnumerableCompat.filter_map. Array#filter_map is 2.7+; the bare call would raise NoMethodError on every module with indexed methods on Ruby 2.5/2.6, get swallowed by the per-mod rescue, and silently drop every FILE scope from initial uploads on those Rubies. Matches the pattern already used at line 692 and 1052 of the same file.
The 3-pass entries -> file_trees -> Array<Scope> pipeline was retained after the per-file streaming index landed, on the rationale that tests exercised the old shape. They do not: no spec calls any of these methods, only four spec comments referenced them by name. Deletes: - collect_extractable_modules (legacy Pass 1; no callers) - group_methods_by_file (called only by collect_extractable_modules) - build_file_trees (legacy Pass 2; no callers) - convert_trees_to_scopes (legacy Pass 3; no callers) Keeps resolves_to_same_module? \u2014 commit 6d52073 ported its call into build_per_file_index, so the helper is live, not legacy. Touch-ups: - Re-point 5 in-file comments (MODULE_SINGLETON_CLASS_PRED docstring, resolves_to_same_module? docstring, SLEEP_EVERY_N_MODULES docstring, build_per_file_index singleton-skip comment, collect_method_names_by_file docstring, convert_tree_to_scope docstring) to the current method names. - Inline the singleton-class skip rationale into build_per_file_index (previously offloaded to collect_extractable_modules). - Re-point 4 spec comments (3 in extractor_spec.rb, 1 in remote_config_integration_spec.rb) to build_per_file_index. Net: -124 LOC in extractor.rb, -4 signatures in extractor.rbs. All 184 spec examples in component_spec + extractor_spec + integration_spec pass; standard + steep clean.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 22755dceb4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The streaming index drops UnboundMethod objects between passes: build_per_file_index (Pass 1) records (mod, method_name, file_path) and build_file_scope (Pass 2) re-resolves the UnboundMethod via mod.instance_method(name). If the method is redefined in another file between the two passes (Rails class reload while extract_all is iterating), Pass 2 gets an UnboundMethod whose source_location is the new file but attributes it to the file_path recorded in Pass 1. The legacy path retained the original UnboundMethod and so couldn't mix files this way. Address review comment by adding a source_location recheck in build_file_scope. After resolving, compare method.source_location[0] to the recorded file_path; drop the entry if they no longer match. When every recorded method for a module has moved out of file_path, drop the module entry entirely so the FILE scope doesn't carry an empty CLASS/MODULE node at a location the module no longer lives in. The hot-load TracePoint enqueues the redefined class and the next debounce window extracts it under the new file_path. Test added: build_file_scope receives a (mod, name) tuple whose method source_location no longer matches the file_path; the FILE scope collapses to nil rather than emitting a CLASS node at the stale location. Verified locally: - bundle exec rspec spec/datadog/symbol_database/: 359 examples, 0 failures - bundle exec steep check lib/datadog/symbol_database/extractor.rb: clean Co-Authored-By: Claude <[email protected]>
The Pass 1 \u2192 Pass 2 stale-method recheck added in c0771a8 uses `next if method_names.any? && method_infos.empty?` to drop module entries whose every recorded method has moved out of file_path. The `method_names.any?` qualifier is what keeps namespace-only modules (empty method-name list via the find_source_file fallback) from being incorrectly dropped. Add a regression test exercising the namespace-only path through build_file_scope. Without the `method_names.any?` qualifier this test would fail because an empty method_names list also produces an empty method_infos list, and the guard would skip the module entry. Verified: - bundle exec rspec spec/datadog/symbol_database/{component,extractor,integration}_spec.rb: 186 examples, 0 failures - bundle exec rake standard typecheck: clean
Replaces ObjectSpace.each_object(Module) with a const-walk rooted at Object that follows Module#constants(false) + const_get(sym, false) with a per-segment autoload? guard. Per @eregon's suggestion on #5872: const-walk reachability is the leak filter by construction — anything reached IS the binding at that name — so resolves_to_same_module? (the post-hoc filter the legacy ObjectSpace path needs) is unnecessary here. Coverage equivalence vs ObjectSpace.each_object(Module): - Const-reachable modules: yielded by both. - Anonymous modules (no Module#name): not yielded; safe_mod_name filtered these out anyway. - Singleton classes: not yielded; legacy code skipped via MODULE_SINGLETON_CLASS_PRED. - "Leaked" classes whose const was remove_const-ed: not yielded; legacy code filtered via resolves_to_same_module?. Autoload safety: const_get(sym, false) still triggers autoload on a pending name. Each segment is guarded with autoload? first (with the same Ruby 2.7+ inherit-false / 2.5-2.6 inherited-included branch used in resolves_to_same_module?), so pending autoloads abort descent into that subtree without triggering customer code or raising LoadError. The legacy collect_extractable_modules still uses ObjectSpace.each_object — it is the path Component#extract_and_upload took prior to #5883's block-form migration and is retained as long as the non-block extract_all API exists. Verified: - bundle exec rspec spec/datadog/symbol_database/extractor_spec.rb (134 examples, 0 failures) — including the existing leak tests (`autoload registered after remove_const`, `class detached from its constant via remove_const`, `subclass with constant of same name as ancestor pending autoload`) which now exercise the const-walk path. - bundle exec rspec spec/datadog/symbol_database/ (362 examples, 0 failures). - bundle exec steep check lib/datadog/symbol_database/extractor.rb (clean). - bundle exec standardrb (clean on changed files). Co-Authored-By: Claude <[email protected]>
What does this PR do?
Refactors
Extractor#extract_allso that the per-call peak working set during symbol extraction is bounded by per-file work, not by total user-class count. Measured steady-state per-call overhead is flat at ~1.1 MB from 1,500 to 8,000 user classes, vs. up to ~160 MB previously.Three changes that compose:
extract_allaccepts an optional block; when given, FILE scopes are yielded one at a time and the per-file working set is released as it is yielded. Without a block,extract_allstill returnsArray<Scope>(test/benchmark API preserved).Module#instance_methods(false)already returns public + protected.group_methods_by_file(legacy) and the newcollect_method_names_by_fileiterateinstance_methods+private_instance_methodsonly — no merged-array allocation, nouniq!pass.collect_extractable_modules → build_file_trees → convert_trees_to_scopesis replaced with a 2-passbuild_per_file_index → build_file_scopeflow. Pass 1 walksObjectSpace.each_object(Module)once and records only(mod_name, Module-ref, [method_name_symbol, ...])per file — noUnboundMethodobjects retained between passes. Pass 2 processes one file at a time, resolvingUnboundMethods just-in-time and releasing them as the FILE scope is built. Combined with the block form, the index is drained destructively (index.shift) so each per-file entry becomes collectable as soon as its FILE scope is consumed.Output shape is unchanged. All existing
extract_allspecs pass; only one spec change — the test that stubbedcollect_extractable_moduleswas updated to stubbuild_per_file_index(the new entry point).Component#extract_and_uploadnow uses the block form and accumulatesextracted_count+targetable_countinline rather than walking the full Array afterward.Motivation:
The original
Extractor#extract_allpeaked at ~74 MB on a 2,500-user-class workload and the apparent peak grew with workspace size. The peak driver was theentries → file_trees → Array<Scope>triple that all coexisted at the start of Pass 3, with every module'sUnboundMethodobjects retained throughout.The new per-file index stores Symbol method names and Module refs only; UnboundMethods are resolved inside one file's build loop where they can be GC'd promptly. The streaming yield removes the
Array<Scope>accumulator from production paths.Memory measurements
There are three distinct memory components, and earlier single-shot RSS measurements conflated all three. Calling them out separately:
1. Per-call peak working set (this PR addresses this)
The extra RSS allocated during one
extract_allcall, measured in steady state (after Ruby's GC heap has reached its working size for this workspace). This is what the user-perceived cost of "running an extraction" actually is.Measured via 5 ms cadence RSS sampling on
taskset -c 0,1,intel_pstate/no_turbo=1, Ruby 3.2.3, three warmup runs to stabilize the heap, then one measured call. Same fixture asbenchmarks/symbol_database_extraction.rb(3 methods per class, one constant).Flat. The per-call cost is now bounded by per-file work (the largest single file's method count + one FILE scope being built and yielded), not by workspace size.
2. One-time GC heap expansion
The first time a process extracts at a given workspace size, Ruby's heap grows from "size needed to hold the loaded classes" to "size needed for that plus the extractor's working set." This happens once over the first 1-2 calls and then stabilizes. Subsequent calls reuse the existing pages.
Measured at 8k classes:
The +33 MB process-lifetime overhead at 8k classes is a one-time cost, not per-call.
GC.stat[:heap_live_slots]matches exactly across runs 2-10.3. Workspace baseline RSS
Grows with class count because Ruby holds the class definitions themselves — method tables, constant slots, iseqs. Visible as the post-class-load RSS reading (82,940 KB at 8k classes vs ~56,000 KB at 1.5k classes). Not extraction overhead; present whether
extract_allever runs or not.Comparison to the previous implementation
The single-shot benchmark in #5798 measures (1) + (2) + (3) together for the first extraction call. That's why pre-PR numbers looked like ~74 MB at 2,500 and ~160 MB at 8,000 — they were dominated by (2), the one-time GC heap expansion to accommodate a peak working set of tens of megabytes.
The pre-PR per-call working set (component 1) was large (~30-60 MB) because of the three-structure retention. After this PR, it's 1.1 MB at every measured size. The headline "−85%" first-call number from earlier drafts of this PR description was real but partly captured by the smaller heap expansion (Ruby needed fewer pages because the working set is smaller); the more important property is that the per-call number is now a constant, not a function of workspace size.
GC.startis not called from production code.Change log entry
Yes. Reduced peak memory usage during Symbol Database extraction on applications with many classes. The per-call working set is now bounded regardless of application size.
Additional Notes:
This PR is layered:
SymDB: stream extract_all + skip redundant visibility scan) — initial streaming yield + visibility-scan skip.di_instrument: reorder header table to match execution order) — unrelated DI benchmark header touch that landed on the same branch; harmless and small.SymDB: per-file streaming index to bound peak memory) — the structural change above.The benchmark used to measure these numbers is
benchmarks/symbol_database_extraction.rb, which lives in #5798. The benchmark's existing methodology (single-shot, no warmup) captures first-call cost (components 1+2+3 from above); use the inline driver in "How to test" below to measure steady-state per-call cost in isolation.How to test the change?
Spec suite (178 examples across component_spec + extractor_spec + integration_spec, all pass locally):
bundle exec rspec spec/datadog/symbol_database/component_spec.rb \ spec/datadog/symbol_database/extractor_spec.rb \ spec/datadog/symbol_database/integration_spec.rbFirst-call memory measurement (requires #5798's benchmark file applied on top):
taskset -c 2,3 bundle exec ruby benchmarks/symbol_database_extraction.rbSteady-state per-call measurement (warm the heap, then sample one call):
Expected output: ~1100 KB at any class count from 1.5k to 8k+.