Skip to content

SymDB: bound Extractor#extract_all per-call peak working set#5883

Merged
p-datadog merged 9 commits into
masterfrom
symdb-extractor-memory-reduction
Jun 11, 2026
Merged

SymDB: bound Extractor#extract_all per-call peak working set#5883
p-datadog merged 9 commits into
masterfrom
symdb-extractor-memory-reduction

Conversation

@p-datadog

@p-datadog p-datadog commented Jun 9, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Refactors Extractor#extract_all so that the per-call peak working set during symbol extraction is bounded by per-file work, not by total user-class count. Measured steady-state per-call overhead is flat at ~1.1 MB from 1,500 to 8,000 user classes, vs. up to ~160 MB previously.

Three changes that compose:

  1. Streaming yield. extract_all accepts an optional block; when given, FILE scopes are yielded one at a time and the per-file working set is released as it is yielded. Without a block, extract_all still returns Array<Scope> (test/benchmark API preserved).
  2. Skip redundant visibility scan. Module#instance_methods(false) already returns public + protected. group_methods_by_file (legacy) and the new collect_method_names_by_file iterate instance_methods + private_instance_methods only — no merged-array allocation, no uniq! pass.
  3. Per-file streaming index. The 3-pass collect_extractable_modules → build_file_trees → convert_trees_to_scopes is replaced with a 2-pass build_per_file_index → build_file_scope flow. Pass 1 walks ObjectSpace.each_object(Module) once and records only (mod_name, Module-ref, [method_name_symbol, ...]) per file — no UnboundMethod objects retained between passes. Pass 2 processes one file at a time, resolving UnboundMethods just-in-time and releasing them as the FILE scope is built. Combined with the block form, the index is drained destructively (index.shift) so each per-file entry becomes collectable as soon as its FILE scope is consumed.

Output shape is unchanged. All existing extract_all specs pass; only one spec change — the test that stubbed collect_extractable_modules was updated to stub build_per_file_index (the new entry point). Component#extract_and_upload now uses the block form and accumulates extracted_count + targetable_count inline rather than walking the full Array afterward.

Motivation:

The original Extractor#extract_all peaked at ~74 MB on a 2,500-user-class workload and the apparent peak grew with workspace size. The peak driver was the entries → file_trees → Array<Scope> triple that all coexisted at the start of Pass 3, with every module's UnboundMethod objects retained throughout.

The new per-file index stores Symbol method names and Module refs only; UnboundMethods are resolved inside one file's build loop where they can be GC'd promptly. The streaming yield removes the Array<Scope> accumulator from production paths.

Memory measurements

There are three distinct memory components, and earlier single-shot RSS measurements conflated all three. Calling them out separately:

1. Per-call peak working set (this PR addresses this)

The extra RSS allocated during one extract_all call, measured in steady state (after Ruby's GC heap has reached its working size for this workspace). This is what the user-perceived cost of "running an extraction" actually is.

Measured via 5 ms cadence RSS sampling on taskset -c 0,1, intel_pstate/no_turbo=1, Ruby 3.2.3, three warmup runs to stabilize the heap, then one measured call. Same fixture as benchmarks/symbol_database_extraction.rb (3 methods per class, one constant).

Classes Steady-state per-call peak (this PR, block form)
1,500 1.1 MB
2,500 1.1 MB
3,500 1.1 MB
5,000 1.1 MB
8,000 1.1 MB

Flat. The per-call cost is now bounded by per-file work (the largest single file's method count + one FILE scope being built and yielded), not by workspace size.

2. One-time GC heap expansion

The first time a process extracts at a given workspace size, Ruby's heap grows from "size needed to hold the loaded classes" to "size needed for that plus the extractor's working set." This happens once over the first 1-2 calls and then stabilizes. Subsequent calls reuse the existing pages.

Measured at 8k classes:

after class load + GC:             82,940 KB
after run 1:                      105,720 KB   (heap pages allocated)
after run 2:                      116,048 KB   (final pages allocated)
after run 3 through run 10:       116,052 KB   (stable, zero further growth)

The +33 MB process-lifetime overhead at 8k classes is a one-time cost, not per-call. GC.stat[:heap_live_slots] matches exactly across runs 2-10.

3. Workspace baseline RSS

Grows with class count because Ruby holds the class definitions themselves — method tables, constant slots, iseqs. Visible as the post-class-load RSS reading (82,940 KB at 8k classes vs ~56,000 KB at 1.5k classes). Not extraction overhead; present whether extract_all ever runs or not.

Comparison to the previous implementation

The single-shot benchmark in #5798 measures (1) + (2) + (3) together for the first extraction call. That's why pre-PR numbers looked like ~74 MB at 2,500 and ~160 MB at 8,000 — they were dominated by (2), the one-time GC heap expansion to accommodate a peak working set of tens of megabytes.

The pre-PR per-call working set (component 1) was large (~30-60 MB) because of the three-structure retention. After this PR, it's 1.1 MB at every measured size. The headline "−85%" first-call number from earlier drafts of this PR description was real but partly captured by the smaller heap expansion (Ruby needed fewer pages because the working set is smaller); the more important property is that the per-call number is now a constant, not a function of workspace size.

GC.start is not called from production code.

Change log entry

Yes. Reduced peak memory usage during Symbol Database extraction on applications with many classes. The per-call working set is now bounded regardless of application size.

Additional Notes:

This PR is layered:

  • Commit 1 (SymDB: stream extract_all + skip redundant visibility scan) — initial streaming yield + visibility-scan skip.
  • Commit 2 (di_instrument: reorder header table to match execution order) — unrelated DI benchmark header touch that landed on the same branch; harmless and small.
  • Commit 3 (SymDB: per-file streaming index to bound peak memory) — the structural change above.

The benchmark used to measure these numbers is benchmarks/symbol_database_extraction.rb, which lives in #5798. The benchmark's existing methodology (single-shot, no warmup) captures first-call cost (components 1+2+3 from above); use the inline driver in "How to test" below to measure steady-state per-call cost in isolation.

How to test the change?

Spec suite (178 examples across component_spec + extractor_spec + integration_spec, all pass locally):

bundle exec rspec spec/datadog/symbol_database/component_spec.rb \
                  spec/datadog/symbol_database/extractor_spec.rb \
                  spec/datadog/symbol_database/integration_spec.rb

First-call memory measurement (requires #5798's benchmark file applied on top):

taskset -c 2,3 bundle exec ruby benchmarks/symbol_database_extraction.rb

Steady-state per-call measurement (warm the heap, then sample one call):

# tools/symdb_steady_state.rb
3.times { extractor.extract_all { |_| } ; GC.start }   # warmup
GC.start
peak = baseline = `ps -o rss= -p #{Process.pid}`.to_i
stop = false
sampler = Thread.new { until stop; cur = `ps -o rss= -p #{Process.pid}`.to_i; peak = cur if cur > peak; sleep 0.005; end }
extractor.extract_all { |_| }
stop = true; sampler.join
puts "steady_overhead_kb=#{peak - baseline}"

Expected output: ~1100 KB at any class count from 1.5k to 8k+.

Two changes that together reduce Extractor#extract_all peak memory from
74 MB to 61 MB on a 2500-user-class workload (-17.6%, measured via
benchmarks/symbol_database_extraction.rb from #5798), with per-iteration
wall time within noise.

Streaming yield (extractor.rb, component.rb)

extract_all now accepts an optional block. When a block is given, FILE
scopes are yielded one at a time and the per-file tree node is shifted
out of the in-progress hash as it is yielded, so the caller can free
each scope after consuming it without retaining the full Array<Scope>.
The entries hash from Pass 1 is also dropped before Pass 3 begins, so
its UnboundMethod references become eligible for collection at the
next GC instead of staying alive for the whole Pass 3 conversion.

Without a block, extract_all still returns the full Array<Scope> as
before \u2014 preserves the test/benchmark API.

Component.extract_and_upload uses the block form and accumulates the
extracted_count + targetable_count inline rather than walking the full
array after extraction. count_targetable_methods was split so the
per-scope helper count_targetable_methods_in_scope is reusable in the
streaming loop (the wrapper that takes the full Array is kept).

Skip redundant visibility scan (extractor.rb)

Module#instance_methods(false) already returns both public and
protected methods. group_methods_by_file was iterating
instance_methods + protected_instance_methods + private_instance_methods
with a uniq! to drop the duplicated protected entries. Iterating only
instance_methods + private_instance_methods covers all three
visibilities without the redundant scan or the merged-array
allocation.
@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jun 9, 2026
@dd-octo-sts dd-octo-sts Bot added the debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database) label Jun 9, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 14 steep:ignore comments, and clears 13 steep:ignore comments.

steep:ignore comments (+14-13)Introduced:
lib/datadog/symbol_database/component.rb:573
lib/datadog/symbol_database/extractor.rb:350
lib/datadog/symbol_database/extractor.rb:362
lib/datadog/symbol_database/extractor.rb:376
lib/datadog/symbol_database/extractor.rb:521
lib/datadog/symbol_database/extractor.rb:632
lib/datadog/symbol_database/extractor.rb:662
lib/datadog/symbol_database/extractor.rb:787
lib/datadog/symbol_database/extractor.rb:864
lib/datadog/symbol_database/extractor.rb:882
lib/datadog/symbol_database/extractor.rb:883
lib/datadog/symbol_database/extractor.rb:962
lib/datadog/symbol_database/extractor.rb:963
lib/datadog/symbol_database/extractor.rb:1011
Cleared:
lib/datadog/symbol_database/component.rb:570
lib/datadog/symbol_database/extractor.rb:306
lib/datadog/symbol_database/extractor.rb:318
lib/datadog/symbol_database/extractor.rb:332
lib/datadog/symbol_database/extractor.rb:477
lib/datadog/symbol_database/extractor.rb:588
lib/datadog/symbol_database/extractor.rb:618
lib/datadog/symbol_database/extractor.rb:740
lib/datadog/symbol_database/extractor.rb:804
lib/datadog/symbol_database/extractor.rb:805
lib/datadog/symbol_database/extractor.rb:885
lib/datadog/symbol_database/extractor.rb:886
lib/datadog/symbol_database/extractor.rb:934

Untyped methods

This PR introduces 4 partially typed methods, and clears 7 partially typed methods. It increases the percentage of typed methods from 65.13% to 65.27% (+0.14%).

Partially typed methods (+4-7)Introduced:
sig/datadog/symbol_database/extractor.rbs:30
└── def convert_tree_to_scope: (::String file_path, untyped root) -> Scope
sig/datadog/symbol_database/extractor.rbs:51
└── def build_class_language_specifics: (Class klass) -> Hash[::Symbol, untyped]
sig/datadog/symbol_database/extractor.rbs:65
└── def place_in_tree: (untyped root, Array[String] name_parts, Module mod, String mod_name, Array[untyped] methods, String file_path) -> void
sig/datadog/symbol_database/extractor.rbs:69
└── def convert_node_to_scope: (untyped node) -> Scope
Cleared:
sig/datadog/symbol_database/extractor.rbs:41
└── def build_class_language_specifics: (Class klass) -> Hash[::Symbol, untyped]
sig/datadog/symbol_database/extractor.rbs:55
└── def collect_extractable_modules: () -> Hash[String, untyped]
sig/datadog/symbol_database/extractor.rbs:57
└── def group_methods_by_file: (Module mod) -> Hash[String, Array[untyped]]
sig/datadog/symbol_database/extractor.rbs:59
└── def build_file_trees: (Hash[String, untyped] entries) -> Hash[String, untyped]
sig/datadog/symbol_database/extractor.rbs:61
└── def place_in_tree: (untyped root, Array[String] name_parts, Module mod, String mod_name, Array[untyped] methods, String file_path) -> void
sig/datadog/symbol_database/extractor.rbs:65
└── def convert_trees_to_scopes: (Hash[String, untyped] file_trees) -> Array[Scope]
sig/datadog/symbol_database/extractor.rbs:67
└── def convert_node_to_scope: (untyped node) -> Scope

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jun 9, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 7.14%
Overall Coverage: 89.84% (-0.05%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5315293 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 9, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-10 18:58:07

Comparing candidate commit c0771a8 in PR branch symdb-extractor-memory-reduction with baseline commit 219cdcb in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Refactor extract_all from a 3-pass collect/build/convert that retained
every module's UnboundMethods until the final conversion, into a 2-pass
index/build flow that holds at most one file's working set at a time.

Pass 1 (new build_per_file_index):
- Walks ObjectSpace.each_object(Module) once.
- For each user-code module, records (file_path, module_ref, method_name_symbols)
  per file the module contributes to.
- Stores Symbol method names plus Module references only. No UnboundMethod
  objects retained between passes. The UnboundMethods allocated to read
  source_location become garbage as the inner loop ends.

Pass 2 (new build_file_scope):
- For each file in the index, resolves UnboundMethods just-in-time for the
  modules that contribute to it and builds the FILE scope with full nested
  MODULE/CLASS hierarchy (same output as the old extract_all).
- When extract_all is called with a block, the index is drained
  destructively (index.shift), so each per-file entry becomes collectable
  as soon as its FILE scope is yielded and consumed.

Output shape unchanged. All existing extract_all specs pass (the one
that stubbed collect_extractable_modules was updated to stub
build_per_file_index instead).

Measured memory effect (taskset -c 0,1, no_turbo=1, vs PR baseline +
the earlier A+D commit):

  Classes  no-block(KB)   block-streaming(KB)
  1500     20,308         ~6,000   (-85% vs original A baseline)
  2500     41,592         ~10,700  (-85%)
  3500     50,032         ~12,900  (-85%)
  5000     71,060         ~16,400  (-83%)
  8000    107,996         ~23,300  (-85%)

The block-form (used by Component) now scales at ~2.3 MB per 1000
additional user classes, down from ~14 MB previously. The remaining
growth is mostly the Ruby runtime's own per-class footprint (method
tables, constant slots) that the extractor walks but does not allocate
\u2014 it is also visible in baseline RSS growth across the same sizes.

The legacy collect_extractable_modules / build_file_trees /
convert_trees_to_scopes / group_methods_by_file methods are kept for
tests that exercise the old shape; extract_all no longer uses them.
@p-datadog p-datadog changed the title SymDB: stream extract_all + skip redundant visibility scan SymDB: bound Extractor#extract_all peak memory via per-file streaming index Jun 9, 2026
@p-datadog p-datadog changed the title SymDB: bound Extractor#extract_all peak memory via per-file streaming index SymDB: bound Extractor#extract_all per-call peak working set Jun 9, 2026
@p-datadog
p-datadog requested a review from Copilot June 9, 2026 21:46
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ae28f1eb8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/symbol_database/extractor.rb Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the SymDB extraction pipeline to reduce peak per-call memory usage by switching Extractor#extract_all to a per-file streaming approach (optional block form) and by avoiding allocations during method visibility scanning.

Changes:

  • Added a block/streaming form for Extractor#extract_all, yielding FILE scopes one at a time while preserving the array-returning legacy behavior when no block is given.
  • Replaced the prior multi-structure extraction flow with a per-file index + per-file scope build path, resolving UnboundMethods just-in-time.
  • Updated Component#extract_and_upload and relevant specs to use the streaming/yield form and accumulate counts during streaming.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
lib/datadog/symbol_database/extractor.rb Implements per-file index + per-file scope build and adds streaming block form to bound working set.
lib/datadog/symbol_database/component.rb Switches initial extraction to streaming consumption and computes counts inline without retaining the full scope array.
spec/datadog/symbol_database/extractor_spec.rb Updates a stub to match the new extraction entry point (build_per_file_index).
spec/datadog/symbol_database/component_spec.rb Updates expectations/stubs to match streaming extract_all behavior (yielding scopes).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/datadog/symbol_database/extractor.rb Outdated
Comment thread lib/datadog/symbol_database/extractor.rb Outdated
p-ddsign and others added 4 commits June 9, 2026 18:20
Apply review fixes to the per-file streaming index work:

- Fix shadowing-outer-local-variable warning at extract_all (Ruby 2.5/2.6)
  by switching the block-form drain to while(pair = shift) + pair[0]/pair[1]
  indexing so the else-branch's |path, file_entries| block params have no
  outer-scope binding to shadow.

- Update RBS signatures: extract_all now accepts an optional block and
  returns nil in block form; add declarations for build_per_file_index,
  collect_method_names_by_file, build_file_scope, convert_tree_to_scope,
  count_targetable_methods_in_scope; remove the now-dead
  count_targetable_methods RBS entry.

- Remove count_targetable_methods (dead after extract_and_upload moved to
  inline per-scope accumulation).

- Add trailing commas in build_file_scope's root hash and
  convert_tree_to_scope's Scope.new call.

- Rewrite the extract_all header comment to match the new two-pass design
  (build_per_file_index → build_file_scope) instead of the old three-pass
  one.

- Update convert_tree_to_scope's @param doc to note both callers
  (build_file_trees and build_file_scope).

- Add block-form tests in extractor_spec.rb covering: yield-per-FILE,
  return-nil contract, scope equivalence with the non-block form, and
  top-level rescue behavior under block form.

Verified locally:
- bundle exec steep check: no errors
- bundle exec rspec spec/datadog/symbol_database/{extractor,component,integration}_spec.rb:
  181 examples, 0 failures
- ruby -W2 require of extractor.rb: no warnings
…r_map on 2.5/2.6

Two fixes on the per-file streaming index path:

- Port the resolves_to_same_module? guard from collect_extractable_modules
  (added in #5872) over to build_per_file_index. The new production entry
  point was missing this check, so the autoload-registered-after-remove_const
  and remove_const-detached extract_all specs regressed once master was
  merged. The check is the same one-liner between safe_mod_name and
  user_code_module? as in the legacy method.

- Replace bare method_names.filter_map at build_file_scope with
  Core::Utils::EnumerableCompat.filter_map. Array#filter_map is 2.7+; the
  bare call would raise NoMethodError on every module with indexed methods
  on Ruby 2.5/2.6, get swallowed by the per-mod rescue, and silently drop
  every FILE scope from initial uploads on those Rubies. Matches the
  pattern already used at line 692 and 1052 of the same file.
The 3-pass entries -> file_trees -> Array<Scope> pipeline was retained
after the per-file streaming index landed, on the rationale that tests
exercised the old shape. They do not: no spec calls any of these
methods, only four spec comments referenced them by name.

Deletes:

- collect_extractable_modules (legacy Pass 1; no callers)
- group_methods_by_file       (called only by collect_extractable_modules)
- build_file_trees            (legacy Pass 2; no callers)
- convert_trees_to_scopes     (legacy Pass 3; no callers)

Keeps resolves_to_same_module? \u2014 commit 6d52073 ported its call
into build_per_file_index, so the helper is live, not legacy.

Touch-ups:

- Re-point 5 in-file comments (MODULE_SINGLETON_CLASS_PRED docstring,
  resolves_to_same_module? docstring, SLEEP_EVERY_N_MODULES docstring,
  build_per_file_index singleton-skip comment, collect_method_names_by_file
  docstring, convert_tree_to_scope docstring) to the current method names.
- Inline the singleton-class skip rationale into build_per_file_index
  (previously offloaded to collect_extractable_modules).
- Re-point 4 spec comments (3 in extractor_spec.rb, 1 in
  remote_config_integration_spec.rb) to build_per_file_index.

Net: -124 LOC in extractor.rb, -4 signatures in extractor.rbs. All 184
spec examples in component_spec + extractor_spec + integration_spec
pass; standard + steep clean.
@p-datadog
p-datadog requested a review from Copilot June 10, 2026 17:42
@p-datadog
p-datadog marked this pull request as ready for review June 10, 2026 17:42
@p-datadog
p-datadog requested a review from a team as a code owner June 10, 2026 17:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 22755dceb4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/symbol_database/extractor.rb

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comment thread lib/datadog/symbol_database/extractor.rb
The streaming index drops UnboundMethod objects between passes:
build_per_file_index (Pass 1) records (mod, method_name, file_path) and
build_file_scope (Pass 2) re-resolves the UnboundMethod via
mod.instance_method(name). If the method is redefined in another file
between the two passes (Rails class reload while extract_all is iterating),
Pass 2 gets an UnboundMethod whose source_location is the new file but
attributes it to the file_path recorded in Pass 1. The legacy path retained
the original UnboundMethod and so couldn't mix files this way.

Address review comment by adding a source_location recheck in
build_file_scope. After resolving, compare method.source_location[0] to the
recorded file_path; drop the entry if they no longer match. When every
recorded method for a module has moved out of file_path, drop the module
entry entirely so the FILE scope doesn't carry an empty CLASS/MODULE node
at a location the module no longer lives in. The hot-load TracePoint
enqueues the redefined class and the next debounce window extracts it
under the new file_path.

Test added: build_file_scope receives a (mod, name) tuple whose method
source_location no longer matches the file_path; the FILE scope collapses
to nil rather than emitting a CLASS node at the stale location.

Verified locally:
- bundle exec rspec spec/datadog/symbol_database/: 359 examples, 0 failures
- bundle exec steep check lib/datadog/symbol_database/extractor.rb: clean

Co-Authored-By: Claude <[email protected]>
The Pass 1 \u2192 Pass 2 stale-method recheck added in c0771a8 uses
`next if method_names.any? && method_infos.empty?` to drop module
entries whose every recorded method has moved out of file_path. The
`method_names.any?` qualifier is what keeps namespace-only modules
(empty method-name list via the find_source_file fallback) from being
incorrectly dropped.

Add a regression test exercising the namespace-only path through
build_file_scope. Without the `method_names.any?` qualifier this test
would fail because an empty method_names list also produces an empty
method_infos list, and the guard would skip the module entry.

Verified:
- bundle exec rspec spec/datadog/symbol_database/{component,extractor,integration}_spec.rb: 186 examples, 0 failures
- bundle exec rake standard typecheck: clean
@p-datadog
p-datadog merged commit d63f718 into master Jun 11, 2026
597 of 599 checks passed
@p-datadog
p-datadog deleted the symdb-extractor-memory-reduction branch June 11, 2026 14:46
@dd-octo-sts dd-octo-sts Bot added this to the 2.36.0 milestone Jun 11, 2026
p-datadog pushed a commit that referenced this pull request Jun 11, 2026
Replaces ObjectSpace.each_object(Module) with a const-walk rooted at
Object that follows Module#constants(false) + const_get(sym, false) with
a per-segment autoload? guard. Per @eregon's suggestion on #5872:
const-walk reachability is the leak filter by construction — anything
reached IS the binding at that name — so resolves_to_same_module? (the
post-hoc filter the legacy ObjectSpace path needs) is unnecessary here.

Coverage equivalence vs ObjectSpace.each_object(Module):
- Const-reachable modules: yielded by both.
- Anonymous modules (no Module#name): not yielded; safe_mod_name filtered
  these out anyway.
- Singleton classes: not yielded; legacy code skipped via
  MODULE_SINGLETON_CLASS_PRED.
- "Leaked" classes whose const was remove_const-ed: not yielded;
  legacy code filtered via resolves_to_same_module?.

Autoload safety: const_get(sym, false) still triggers autoload on a
pending name. Each segment is guarded with autoload? first (with the
same Ruby 2.7+ inherit-false / 2.5-2.6 inherited-included branch used
in resolves_to_same_module?), so pending autoloads abort descent into
that subtree without triggering customer code or raising LoadError.

The legacy collect_extractable_modules still uses
ObjectSpace.each_object — it is the path Component#extract_and_upload
took prior to #5883's block-form migration and is retained as long as
the non-block extract_all API exists.

Verified:
- bundle exec rspec spec/datadog/symbol_database/extractor_spec.rb
  (134 examples, 0 failures) — including the existing leak tests
  (`autoload registered after remove_const`, `class detached from its
  constant via remove_const`, `subclass with constant of same name as
  ancestor pending autoload`) which now exercise the const-walk path.
- bundle exec rspec spec/datadog/symbol_database/ (362 examples,
  0 failures).
- bundle exec steep check lib/datadog/symbol_database/extractor.rb
  (clean).
- bundle exec standardrb (clean on changed files).

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants