Skip to content

SymDB: TracePoint :class hot-load hook for incremental class extraction#5697

Merged
p-datadog merged 12 commits into
masterfrom
symdb-hot-load-hook
Jun 8, 2026
Merged

SymDB: TracePoint :class hot-load hook for incremental class extraction#5697
p-datadog merged 12 commits into
masterfrom
symdb-hot-load-hook

Conversation

@p-datadog

@p-datadog p-datadog commented May 6, 2026

Copy link
Copy Markdown
Member

JIRA: DEBUG-5367

What does this PR do?

Adds a TracePoint :class hook to the symdb Component so that classes loaded after initial extraction (request-time autoloads, late initializers, plugin systems, STI subclasses on first query, dynamically-created classes) reach the symbol DB. Closes a cross-tracer parity gap with Java (ClassFileTransformer), Python (BaseModuleWatchdog#after_import), and .NET (AppDomain.AssemblyLoad) — all of which combine an initial scan with a hot-load hook for continuous coverage.

Also wires Component#after_fork! into Components#after_fork so that each forked worker has a clean Component state — own initial extraction, own TracePoint, own scheduler thread — matching the existing per-process RC client model.

Motivation:

Ruby's symdb today scans ObjectSpace.each_object(Module) once per process. Anything not loaded by :after_initialize is invisible to the upload — under eager_load=false, that's most of the app; under eager_load=true, that's still anything that loads later (request-time autoloads, plugin code, dynamically-created classes, STI subclasses on first query). Java/Python/.NET don't have this gap because they hook the runtime's class-load mechanism. This PR closes it.

Design background:

  • Cross-tracer comparison and Ruby-native equivalents evaluated in the symdb project research notes.
  • TracePoint :class chosen over Zeitwerk on_load (Rails-only), Module#const_added (Ruby 3.1+ only), and polling (high latency).
  • Per-class incremental extraction via existing Extractor#extract(mod) chosen over full re-scan (matches Java/Python; bandwidth scales with what changed, not total app size).

Behavioral changes (intentional):

  1. Multiple uploads per process. Initial upload + one upload per debounce window of new class loads. Backend dedupes overlapping FILE scopes.
  2. No more class-level once-per-process flag. Component.uploaded_this_process? and the surrounding machinery removed. The two-uploads bug fix that introduced the flag is preserved by the per-instance scheduler debounce — multiple start_upload calls on one Component still collapse into one extraction within the debounce window. Cross-Component dedup is intentionally removed: each Component (e.g. after a Datadog.configure reconfigure) does its own initial extraction, which the new Component needs in order to have a hot-load baseline.
  3. eager_load=false no longer suppresses auto-deferred upload. Under eager_load=false, the initial upload may be small but the hot-load hook covers the rest as the app exercises code.
  4. wait_for_idle semantics unchanged from the script's perspective but the implementation waits on @last_upload_time advance instead of the removed flag.
  5. stop_upload now suppresses the hot-load hook. Disables the TracePoint, clears the hot-load buffer, and resets @initial_extraction_done. Without this, post-disable class loads would keep re-arming the scheduler via enqueue_hot_load — RC could "stop" symdb but extraction continued. A subsequent start_upload (RC re-enable) does a fresh extract_all rather than draining an empty buffer.
  6. Per-fork Component reset (Component#after_fork!). Wired into Core::Configuration::Components#after_fork alongside the existing remote&.after_fork. In a forked child, the parent's enabled TracePoint is copied into the child where it stays rooted by the VM — after_fork! disables the inherited TracePoint before clearing the ivar, reinitializes mutexes and condition variables (orphan-lock guard), and resets the scheduler thread reference. Force-upload mode re-registers the deferred upload in the child. RC mode relies on the child's re-subscribed RC client to call start_upload — each process is its own RC subscriber and its own symbol uploader. Cross-process upload dedup of identical FILE scopes from preload_app! + eager_load=true workers is the backend's responsibility, not the tracer's.

Change log entry

Yes: Symbol database: lazily loaded (those loaded after application boots) classes will now be shown in Debugger UI autocomplete

Additional Notes:

  • Hot-load coverage is on by default; no env-var opt-out for MVP. If telemetry surfaces excessive uploads in some app shape, an opt-out can be added.
  • Open question: backend dedup of overlapping FILE scopes from one process. The agnostic provider's merge-on-query path should handle this, but verifying against staging is on the verification list.
  • The eager_load gate removal (add Rubocop for style guidelines #3) interacts with the two-uploads investigation. The actual fix that resolved that bug was the per-instance scheduler + the gate; with hot-load present, the gate is no longer needed because under-extracted initial uploads self-correct. A dedicated regression test ('debounce regression') exercises the per-instance scheduler half of the original fix to confirm it still holds.
  • Type signatures: Component.rbs is also cleaned up — the stale class-level dedup declarations (uploaded_this_process?, mark_uploaded, etc.) are removed, and @logger / initialize / environment_supported? are typed as SymbolDatabase::Logger (not untyped).

How to test the change?

Automated end-to-end coverage in spec/datadog/symbol_database/remote_config_integration_spec.rb under describe 'hot-load end-to-end (TracePoint :class → buffer → debounce → upload)'; also manually verified via my test app.

@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label May 6, 2026
@pr-commenter

pr-commenter Bot commented May 6, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-04 19:08:18

Comparing candidate commit 8eb5d67 in PR branch symdb-hot-load-hook with baseline commit 2e2d051 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@p-datadog
p-datadog force-pushed the symdb-hot-load-hook branch from efbfd03 to 63abb16 Compare May 11, 2026 14:48
@p-datadog
p-datadog changed the base branch from symbol-database-upload to symbol-database-upload-rebased May 11, 2026 16:27
Base automatically changed from symbol-database-upload-rebased to master May 18, 2026 15:59
@p-datadog
p-datadog force-pushed the symdb-hot-load-hook branch from 63abb16 to ade017a Compare May 18, 2026 16:18
@dd-octo-sts dd-octo-sts Bot added core Involves Datadog core libraries debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database) labels May 18, 2026
@dd-octo-sts

dd-octo-sts Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 1 steep:ignore comment, and clears 1 steep:ignore comment.

steep:ignore comments (+1-1)Introduced:
lib/datadog/symbol_database/component.rb:548
Cleared:
lib/datadog/symbol_database/component.rb:175

p-datadog pushed a commit that referenced this pull request May 19, 2026
CI's Steep check on PR #5697 flagged 4 problems:

1. component.rb:477 - NoMethod on @hot_load_tracepoint.enable. Steep
   does not narrow instance variable types after assignment within the
   same method, so the `(::TracePoint | nil)` RBS type persists past the
   `@hot_load_tracepoint = TracePoint.new(...)` assignment. Added
   `steep:ignore NoMethod` per the project Steep policy: false positives
   from narrowing limitations are silenced, never worked around by
   refactoring working code.

2. component.rb:472 - redundant `# steep:ignore:start` block around the
   TracePoint body. The block contents (`mod.singleton_class?`, `send`)
   do not produce Steep errors. Removed the ignore.

3. remote.rb:106, 113 - redundant `# steep:ignore NoMethod` on
   `change.content` accesses. The union type's variants must now all
   expose `.content`, so Steep no longer flags the access. Removed both
   ignores.

Verified locally: `bundle exec rake typecheck` clean.

Co-Authored-By: Claude <[email protected]>
p-datadog pushed a commit that referenced this pull request Jun 2, 2026
CI's Steep check on PR #5697 flagged 4 problems:

1. component.rb:477 - NoMethod on @hot_load_tracepoint.enable. Steep
   does not narrow instance variable types after assignment within the
   same method, so the `(::TracePoint | nil)` RBS type persists past the
   `@hot_load_tracepoint = TracePoint.new(...)` assignment. Added
   `steep:ignore NoMethod` per the project Steep policy: false positives
   from narrowing limitations are silenced, never worked around by
   refactoring working code.

2. component.rb:472 - redundant `# steep:ignore:start` block around the
   TracePoint body. The block contents (`mod.singleton_class?`, `send`)
   do not produce Steep errors. Removed the ignore.

3. remote.rb:106, 113 - redundant `# steep:ignore NoMethod` on
   `change.content` accesses. The union type's variants must now all
   expose `.content`, so Steep no longer flags the access. Removed both
   ignores.

Verified locally: `bundle exec rake typecheck` clean.

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog force-pushed the symdb-hot-load-hook branch from 9c7dcf0 to 421048c Compare June 2, 2026 20:41
@p-datadog
p-datadog requested a review from Copilot June 2, 2026 20:41
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 421048cc98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/symbol_database/component.rb
Comment thread lib/datadog/symbol_database/component.rb
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jun 2, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 7.89%
Overall Coverage: 90.08% (-7.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a57b116 | Docs | Datadog PR Page | Give us feedback!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds continuous (hot-load) symbol extraction for Ruby SymDB by installing a TracePoint :class hook, enabling incremental uploads for classes loaded after the initial scan, and updates fork handling so child processes reset per-instance threading/mutex/hook state correctly.

Changes:

  • Add per-Component TracePoint :class hook + buffered incremental extraction path (Extractor#extract(mod)) after initial extract_all.
  • Replace class-level “uploaded_this_process” dedup semantics with per-instance debounce scheduling + wait_for_idle based on @last_upload_time.
  • Add Component#after_fork! and invoke it from Core components’ fork hook.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
lib/datadog/symbol_database/component.rb Implements hot-load buffering/extraction, new wait semantics, and fork-state reset.
lib/datadog/core/configuration/components.rb Calls symbol_database&.after_fork! during fork handling.
sig/datadog/symbol_database/component.rbs Updates type surface for new ivars/methods (but needs cleanup to match removed APIs).
spec/datadog/symbol_database/component_spec.rb Adds/adjusts unit coverage for fork reset, debounce regression, and hot-load behavior.
spec/datadog/symbol_database/remote_config_integration_spec.rb Updates integration expectations to align with incremental extraction behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/datadog/symbol_database/component.rb Outdated
Comment thread lib/datadog/symbol_database/component.rb
Comment thread spec/datadog/symbol_database/component_spec.rb Outdated
Comment thread sig/datadog/symbol_database/component.rbs Outdated
Comment thread sig/datadog/symbol_database/component.rbs
Comment thread sig/datadog/symbol_database/component.rbs
Comment thread sig/datadog/symbol_database/component.rbs
p-datadog pushed a commit that referenced this pull request Jun 2, 2026
CI's Steep check on PR #5697 flagged 4 problems:

1. component.rb:477 - NoMethod on @hot_load_tracepoint.enable. Steep
   does not narrow instance variable types after assignment within the
   same method, so the `(::TracePoint | nil)` RBS type persists past the
   `@hot_load_tracepoint = TracePoint.new(...)` assignment. Added
   `steep:ignore NoMethod` per the project Steep policy: false positives
   from narrowing limitations are silenced, never worked around by
   refactoring working code.

2. component.rb:472 - redundant `# steep:ignore:start` block around the
   TracePoint body. The block contents (`mod.singleton_class?`, `send`)
   do not produce Steep errors. Removed the ignore.

3. remote.rb:106, 113 - redundant `# steep:ignore NoMethod` on
   `change.content` accesses. The union type's variants must now all
   expose `.content`, so Steep no longer flags the access. Removed both
   ignores.

Verified locally: `bundle exec rake typecheck` clean.

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog force-pushed the symdb-hot-load-hook branch from 421048c to 09bf180 Compare June 2, 2026 20:47
p-datadog pushed a commit that referenced this pull request Jun 2, 2026
… state

Code (lib/datadog/symbol_database/component.rb):
- stop_upload: disable @hot_load_tracepoint, clear @hot_load_buffer, reset
  @initial_extraction_done. Without this, post-disable class loads kept
  re-arming the scheduler via enqueue_hot_load — RC could "stop" symdb
  but extraction continued every time a class loaded.
- after_fork!: @hot_load_tracepoint&.disable before niling the ivar. In a
  preloaded forking server, fork copies the parent's enabled TracePoint
  into the child where it stays rooted by the VM. Without an explicit
  disable, the child held two enabled TracePoints after the next
  start_upload: the inherited one (now unreferenced and unmanageable)
  plus the freshly installed one. shutdown! could only reach the new
  one.
- after_fork! docstring no longer references an internal-only path.

Specs (spec/datadog/symbol_database/component_spec.rb):
- Add: stop_upload disables the TracePoint, clears the buffer, resets the
  initial-extraction flag, and prevents a post-stop class definition
  from triggering extraction.
- Add: after_fork! disables the inherited TracePoint before clearing
  the reference.
- Rewrite 'does not extract again after start, stop, re-start' as
  're-runs extract_all after stop_upload + start_upload'. With
  @initial_extraction_done reset by stop_upload, a subsequent
  start_upload models RC re-enable and runs a fresh extract_all.
- Delete 'short-circuits when the process has already uploaded' — it
  called the removed mark_uploaded class method and would raise
  NoMethodError at runtime.
- Delete the duplicate stale debounce-named example that still asserted
  eq(1) under the rebuild-then-start sequence — the actual debounce
  property is covered earlier in the file, and 'each Component built
  across reconfigurations extracts independently' covers the new
  per-Component model.

Integration spec (spec/datadog/symbol_database/remote_config_integration_spec.rb):
- Rewrite 'a second start_upload with no new class loads' to omit
  stop_upload between calls. With the new stop_upload semantics, the
  prior shape (start, stop, start) would now extract twice; the
  property the test cares about — empty hot-load drain produces no
  upload — is verified by back-to-back start_upload calls.

Type signatures (sig/datadog/symbol_database/component.rbs):
- Remove stale class-level dedup declarations: @uploaded_this_process,
  @upload_done_mutex, @upload_done_cv, and the four class methods —
  none exist in the implementation any more.
- Restore @logger, initialize logger param, and environment_supported?
  logger param to SymbolDatabase::Logger. Steep is clean with the
  tighter type; the earlier 'untyped' revert was based on a wrong
  reading of the Uploader sig (uploader.rbs already declares
  SymbolDatabase::Logger).

Addresses chatgpt-codex-connector and Copilot review comments on
PR #5697.

Verification: bundle exec rspec spec/datadog/symbol_database/component_spec.rb
→ 42 examples, 0 failures. bundle exec steep check sig/datadog/symbol_database
lib/datadog/symbol_database → clean.

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog marked this pull request as ready for review June 4, 2026 17:13
@p-datadog
p-datadog requested a review from a team as a code owner June 4, 2026 17:13

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe820c7a1b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/symbol_database/component.rb Outdated
Comment thread lib/datadog/symbol_database/component.rb
…op ignore

CI's Steep run flagged 10 errors after the previous "fill in concrete types"
commit. Two type-fill changes were too tight, and one steep:ignore removal
was unverified locally because libdatadog version mismatch blocked Steep
here at the time. CI surfaced all three.

1. Component @logger / initialize logger / environment_supported? logger:
   revert to `untyped`. The tightened `SymbolDatabase::Logger` type didn't
   match Uploader's `Datadog::Core::Logger` parameter type, surfacing a
   long-standing inconsistency between the SymDB Logger wrapper and the
   downstream component sig declarations. Reverting to `untyped` matches
   the established cross-component logger-handoff convention (see DI's
   sigs for the same pattern); fixing the underlying inconsistency in
   Uploader / Extractor / ScopeBatcher sigs is out of scope for this PR.

2. Remote process_change `change` and receiver block's `changes`: revert
   to `untyped`. The union type
   `Repository::Change::Deleted | ::Inserted | ::Updated` looked right but
   Steep cannot narrow the union by the `case change.type` dispatch the
   code uses — `.content` exists only on Inserted/Updated, `.previous`
   only on Deleted/Updated, so every access through the union flagged.
   The runtime dispatch via `.type` symbol matches what the code does
   correctly; reverting to `untyped` matches the pattern. (DI's
   remote.rbs takes the same approach.)

3. scheduler_loop in component.rb:306: reinstate a per-line
   `# steep:ignore NoMethod` on the `@scheduled_at - Time.get_time`
   line, with an inline comment explaining that Steep does not narrow
   nullable instance variables across an `if @scheduled_at.nil?`
   check. Per-line rather than the previously removed block-level
   ignore — only one line needs suppression.

Verification:
- bundle exec steep check → No type error detected. (locally, with
  libdatadog 33.0.0.1.0 now installed)

Co-Authored-By: Claude <[email protected]>
p-ddsign added 2 commits June 4, 2026 13:17
Closes the cross-tracer parity gap: Java (ClassFileTransformer), Python
(BaseModuleWatchdog#after_import), and .NET (AppDomain.AssemblyLoad)
all combine an initial scan with a hot-load hook for continuous coverage
of dynamically loaded code. Ruby's symdb only scanned once, so classes
loaded after :after_initialize (request-time autoloads, late
initializers, plugin systems, STI subclasses on first query, dynamically
created classes) never reached the symbol DB.

Code changes (lib/datadog/symbol_database/component.rb):
- Install per-instance TracePoint :class hook lazily on first start_upload.
  Filters singleton classes (matches extract_all). Pushes loaded modules
  onto a per-instance buffer; signals the scheduler.
- extract_and_upload now distinguishes initial extraction (extract_all,
  full ObjectSpace walk) from incremental (drain buffer, call
  Extractor#extract per unique module). Buffer cleared before initial
  extraction so the post-init drain doesn't reprocess what extract_all
  already covered.
- Disable the TracePoint in shutdown! before the scheduler stops.
- Remove the class-level @uploaded_this_process flag and its three
  short-circuit sites (start_upload + 2 in scheduler_loop). The actual
  two-uploads fix is the per-instance scheduler debounce; the flag was
  belt-and-braces. Removing it allows hot-load extractions after the
  initial upload.
- Remove Component.mark_uploaded, .uploaded_this_process?,
  .reset_uploaded_this_process_for_tests!, @upload_done_mutex,
  @upload_done_cv — all dead with the flag gone.
- Rewrite wait_for_idle to wait on @last_upload_time advance via a
  per-instance condition variable. Preserves the short-lived-script
  semantic (script triggers upload, blocks for it to complete) without
  depending on the one-shot flag.
- Remove the Rails.application.config.eager_load gate in
  schedule_deferred_upload. Under hot-load, an under-extracted initial
  upload self-corrects as the app exercises code.

Spec changes:
- spec/datadog/symbol_database/component_spec.rb: drop the
  reset_uploaded_this_process_for_tests! before-block and the
  short-circuit test; rewrite the two-uploads regression as 'debounce
  collapses bursts of start_upload calls' (the actual property the
  scheduler debounce holds) plus 'each Component built across
  reconfigurations extracts independently' (the new explicit invariant);
  add a hot-load regression test that defines a class via eval after
  initial upload and asserts Extractor#extract is called for it.
- spec/datadog/symbol_database/remote_config_integration_spec.rb: drop
  reset; rewrite 'class-level dedup prevents re-upload' to assert the new
  invariant (a second start_upload with no new class loads produces no
  extra captured upload because the hot-load drain is empty).

Type signatures (sig/datadog/symbol_database/component.rbs): add buffer
ivars, hot-load hook ivars, last_upload_time_cv; remove class-level flag
machinery; add private extract_hot_load_buffer / install_hot_load_hook
/ enqueue_hot_load.

Verification: 316/316 specs pass (full symbol_database/ suite). Steep
clean. StandardRB clean. Two-uploads regression preserved in renamed form
(per-instance debounce continues to collapse bursts within one Component;
cross-Component dedup is intentionally removed).
In a forked child process, only the forking thread survives. Mutexes
and condition variables are copied without ownership tracking (orphan
locks), the scheduler thread reference is dead, and the parent's
TracePoint hook is bound to the dead scheduler. Without an after_fork
reset, the first attempt to acquire a symdb mutex in the child can
deadlock on an orphan lock, and a force_upload-mode child never picks
up where the parent left off.

`Component#after_fork!` reinitializes:

- Hot-load: buffer cleared, `@initial_extraction_done = false`,
  `@hot_load_tracepoint = nil`. The next `start_upload` installs a
  fresh TracePoint bound to the child's component and triggers its own
  initial extraction.
- Scheduler: thread/`@scheduled_at`/`@scheduler_signaled` cleared;
  `@scheduler_mutex` and `@scheduler_cv` reinitialized.
- Upload: `@upload_in_progress = false`; `@mutex`,
  `@upload_in_progress_cv`, `@last_upload_time_cv`,
  `@hot_load_buffer_mutex` reinitialized.
- Force-upload mode: re-registers the deferred-upload callback in the
  child. In Rails, `:after_initialize` already fired in the parent,
  so the on_load block runs immediately and the child schedules its
  own upload.

Cross-process upload deduplication is deliberately out of scope:
each forked Component performs its own initial extraction. Workers
in `preload_app! + eager_load=true` deployments hold identical code
to the parent — backend content dedup is tracked separately in
`projects/symdb/backlog/impl.md §Fork/child dedup` and
`projects/symdb/design/fork-architecture.md`.

Wired from `Core::Configuration::Components#after_fork` alongside
telemetry/remote/crashtracker/ProcessDiscovery.

Unit specs cover: mutex/CV reinit, scheduler thread/state reset,
hot-load buffer/flag/tracepoint reset, `@upload_in_progress`
reset, force-upload re-trigger, and that start_upload still works
after after_fork!.

Local rspec couldn't run (libdatadog 33.0.0.1.0 not installed in
this dev env) — CI is the source of truth.
p-ddsign and others added 3 commits June 4, 2026 13:17
CI's Steep check on PR #5697 flagged 4 problems:

1. component.rb:477 - NoMethod on @hot_load_tracepoint.enable. Steep
   does not narrow instance variable types after assignment within the
   same method, so the `(::TracePoint | nil)` RBS type persists past the
   `@hot_load_tracepoint = TracePoint.new(...)` assignment. Added
   `steep:ignore NoMethod` per the project Steep policy: false positives
   from narrowing limitations are silenced, never worked around by
   refactoring working code.

2. component.rb:472 - redundant `# steep:ignore:start` block around the
   TracePoint body. The block contents (`mod.singleton_class?`, `send`)
   do not produce Steep errors. Removed the ignore.

3. remote.rb:106, 113 - redundant `# steep:ignore NoMethod` on
   `change.content` accesses. The union type's variants must now all
   expose `.content`, so Steep no longer flags the access. Removed both
   ignores.

Verified locally: `bundle exec rake typecheck` clean.

Co-Authored-By: Claude <[email protected]>
The previous Steep fix in 51f692a removed two `# steep:ignore
NoMethod` directives in remote.rb because Steep flagged them as
redundant. But the redundancy was a symptom — the underlying cause was
in `sig/datadog/symbol_database/remote.rbs`, where commit e002f60
weakened `change` and `changes` parameters to `untyped` to bypass union
narrowing issues. With `change: untyped`, `change.content` type-checks
trivially, which made the genuine NoMethod-suppressing ignores look
redundant.

Restore the proper types on remote.rbs (matching master and DI):

- `receiver` block param: `Array[untyped]` →
  `Array[Datadog::Core::Remote::Configuration::Repository::change]`
- `process_change` change param: `untyped` →
  `Datadog::Core::Remote::Configuration::Repository::change`

The `case change.type` body in process_change already uses
`# @type var change: ...Inserted/Updated/Deleted` to narrow within each
when branch. The `else` and `rescue` branches operate on the un-narrowed
union, so reinstate the `# steep:ignore NoMethod` directives there —
they suppress real errors, not redundant ones.

Verification: `bundle exec rake typecheck` clean.

Co-Authored-By: Claude <[email protected]>
… state

Code (lib/datadog/symbol_database/component.rb):
- stop_upload: disable @hot_load_tracepoint, clear @hot_load_buffer, reset
  @initial_extraction_done. Without this, post-disable class loads kept
  re-arming the scheduler via enqueue_hot_load — RC could "stop" symdb
  but extraction continued every time a class loaded.
- after_fork!: @hot_load_tracepoint&.disable before niling the ivar. In a
  preloaded forking server, fork copies the parent's enabled TracePoint
  into the child where it stays rooted by the VM. Without an explicit
  disable, the child held two enabled TracePoints after the next
  start_upload: the inherited one (now unreferenced and unmanageable)
  plus the freshly installed one. shutdown! could only reach the new
  one.
- after_fork! docstring no longer references an internal-only path.

Specs (spec/datadog/symbol_database/component_spec.rb):
- Add: stop_upload disables the TracePoint, clears the buffer, resets the
  initial-extraction flag, and prevents a post-stop class definition
  from triggering extraction.
- Add: after_fork! disables the inherited TracePoint before clearing
  the reference.
- Rewrite 'does not extract again after start, stop, re-start' as
  're-runs extract_all after stop_upload + start_upload'. With
  @initial_extraction_done reset by stop_upload, a subsequent
  start_upload models RC re-enable and runs a fresh extract_all.
- Delete 'short-circuits when the process has already uploaded' — it
  called the removed mark_uploaded class method and would raise
  NoMethodError at runtime.
- Delete the duplicate stale debounce-named example that still asserted
  eq(1) under the rebuild-then-start sequence — the actual debounce
  property is covered earlier in the file, and 'each Component built
  across reconfigurations extracts independently' covers the new
  per-Component model.

Integration spec (spec/datadog/symbol_database/remote_config_integration_spec.rb):
- Rewrite 'a second start_upload with no new class loads' to omit
  stop_upload between calls. With the new stop_upload semantics, the
  prior shape (start, stop, start) would now extract twice; the
  property the test cares about — empty hot-load drain produces no
  upload — is verified by back-to-back start_upload calls.

Type signatures (sig/datadog/symbol_database/component.rbs):
- Remove stale class-level dedup declarations: @uploaded_this_process,
  @upload_done_mutex, @upload_done_cv, and the four class methods —
  none exist in the implementation any more.
- Restore @logger, initialize logger param, and environment_supported?
  logger param to SymbolDatabase::Logger. Steep is clean with the
  tighter type; the earlier 'untyped' revert was based on a wrong
  reading of the Uploader sig (uploader.rbs already declares
  SymbolDatabase::Logger).

Addresses chatgpt-codex-connector and Copilot review comments on
PR #5697.

Verification: bundle exec rspec spec/datadog/symbol_database/component_spec.rb
→ 42 examples, 0 failures. bundle exec steep check sig/datadog/symbol_database
lib/datadog/symbol_database → clean.

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog force-pushed the symdb-hot-load-hook branch from fe820c7 to 6935e30 Compare June 4, 2026 17:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comment thread lib/datadog/symbol_database/component.rb Outdated
Comment thread lib/datadog/symbol_database/component.rb
p-ddsign and others added 6 commits June 4, 2026 14:06
Address review comment from chatgpt-codex-connector: the :class hook
called `mod.singleton_class?` directly. User code defining
`def self.singleton_class?(arg)` (incompatible signature, or any other
override) would raise inside the TracePoint and abort the user's class
definition on reopen.

- Added `MODULE_SINGLETON_CLASS_PRED` constant in
  `lib/datadog/symbol_database/component.rb` mirroring the same defense
  in `lib/datadog/symbol_database/extractor.rb:84` and dispatched via
  `MODULE_SINGLETON_CLASS_PRED.bind(mod).call`.
- Added a spec in `spec/datadog/symbol_database/component_spec.rb` that
  reopens a class with `def self.singleton_class?(_arg)` and asserts
  the hook does not raise and still enqueues the module.

Verified: spec/datadog/symbol_database/ (347 examples, 0 failures);
Steep clean; StandardRB clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
…eduped

Address review comment from chatgpt-codex-connector:
`@scope_batcher` was the one piece of fork-affected state that
`after_fork!` left inherited. `ScopeBatcher#add_scope`
(`lib/datadog/symbol_database/scope_batcher.rb:89`) skips any scope
whose name is already in `@uploaded_modules`. A child re-extracting
under `preload_app!` therefore dropped every scope name the parent had
already uploaded — silently, with no log line — and the child's
extract_all produced an empty upload.

- Replaced `@scope_batcher` with a fresh `ScopeBatcher` instance in
  `lib/datadog/symbol_database/component.rb#after_fork!`, alongside the
  other fork-local reinit. Also gives the child a fresh batcher mutex
  and timer-thread reference.
- Extended the `after_fork!` doc comment to list the batcher with the
  other recreated state and explain the dedup-set hazard.
- Added a spec in `spec/datadog/symbol_database/component_spec.rb` that
  verifies `after_fork!` replaces `@scope_batcher` with a fresh
  instance.

Verified: spec/datadog/symbol_database/ (348 examples, 0 failures);
Steep clean; StandardRB clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
Address two Copilot review comments on the same race pattern:
`stop_upload` (lines 215-216) and `shutdown!` (lines 255-256) both
mutated `@hot_load_tracepoint` outside `@scheduler_mutex`, while
`start_upload#install_hot_load_hook` installs/enables it inside the
mutex. The leaking interleaving:

1. T1 (stop_upload or shutdown!) reads `@hot_load_tracepoint` as nil
   or disabled.
2. T2 (start_upload) acquires `@scheduler_mutex`, installs and enables
   a fresh TracePoint, exits the mutex.
3. T1's `@hot_load_tracepoint = nil` clears the reference, but the
   newly enabled TracePoint stays rooted by the VM.

For `stop_upload`, the leaked TracePoint keeps re-scheduling uploads
after RC disabled symdb. For `shutdown!`, `enqueue_hot_load`'s
`return if @shutdown` skips re-scheduling but only after the buffer
push — `@hot_load_buffer` grows unbounded for the rest of the process
lifetime.

- Moved `@hot_load_tracepoint&.disable; @hot_load_tracepoint = nil`
  inside the existing `@scheduler_mutex.synchronize` block in both
  methods, so the TracePoint teardown is atomic with the scheduler
  state mutation against a concurrent `start_upload`.
- Updated the doc comments on both methods to state the locking
  discipline.
- Added a regression test verifying a class load after `shutdown!`
  does not enqueue into the hot-load buffer (parallel to the existing
  post-`stop_upload` coverage).

`after_fork!` also mutates `@hot_load_tracepoint` outside the mutex,
but only the calling thread is alive in the child (fork invariant),
so there is no race — left as-is.

Verified: spec/datadog/symbol_database/ (348 examples, 0 failures);
Steep clean; StandardRB clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
Steep CI failed with `Cannot find the declaration of constant:
MODULE_SINGLETON_CLASS_PRED` at lib/datadog/symbol_database/component.rb:54
and :545 — the constant was added in 1fc608e to harden the hot-load
TracePoint against user `singleton_class?` overrides, but the matching
RBS declaration was not added.

- Added `MODULE_SINGLETON_CLASS_PRED: ::UnboundMethod` next to the
  existing `EXTRACT_DEBOUNCE_INTERVAL` declaration in
  sig/datadog/symbol_database/component.rbs, mirroring the
  Extractor::MODULE_SINGLETON_CLASS_PRED declaration in
  sig/datadog/symbol_database/extractor.rbs.

Verified locally: `bundle exec steep check` clean;
`bundle exec rspec spec/datadog/symbol_database/` 348 examples 0 failures;
`bundle exec rake standard` clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
Five-step verification of the hot-load hook in the existing
remote_config_integration_spec — initial upload runs, asserts the
runtime-defined class is not in the initial payload, defines the class
via 'load' so source_location passes user_code_path?, waits for the
debounced second upload event-driven via 'wait_for_idle' (no sleep),
asserts the class is present in the second upload's gunzipped payload.

Complements the existing component_spec hot-load coverage test which
mocks Extractor#extract — this test exercises the full tracer flow
through the transport boundary.
Test was measured at ~0.25s locally; 30s was cargo-culted from
surrounding tests in the same file. 5s gives ~20x headroom over
measured runtime while letting a broken signal fail in seconds
instead of a half-minute. Added a code comment recording the
measurement.
@p-datadog
p-datadog merged commit 3ff4f70 into master Jun 8, 2026
591 checks passed
@p-datadog
p-datadog deleted the symdb-hot-load-hook branch June 8, 2026 18:45
@dd-octo-sts dd-octo-sts Bot added this to the 2.36.0 milestone Jun 8, 2026
p-datadog pushed a commit that referenced this pull request Jun 8, 2026
The hot-load end-to-end test in remote_config_integration_spec.rb failed
on Ruby 2.6 batch 0 with seed 16392 — two test failures cascading:

  1) hot-load end-to-end / uploads a class defined after the initial
     upload completes:
     `expect(initial_form_count).to be >= 1` got 0
  2) extract_all module with methods AND nested class in same file
     places child class under parent module in the same FILE scope:
     `expect(file_scope).not_to be_nil`

Root cause of (1): the test called `wait_for_idle(timeout: 5)` without
checking the return value. On Ruby 2.6 with heavy monkey-patching from
AppSec specs running in seed-16392 order before this test, extract_all
exceeded 5s — the extractor.rb comment at collect_extractable_modules
specifically calls this out:

  > on Ruby 2.6 specifically, Module#name on unnamed singleton classes
  > with long ancestor chains (e.g. through monkey-patches prepended
  > into Kernel, common in dd-trace-rb test processes) is O(ancestors)
  > — measured ~20ms per call

wait_for_idle silently returned false on timeout, captured_forms was
still empty, the assertion failed at line 235. The test body's
`component.shutdown!` at the end was never reached → scheduler thread
leaked → continued running extract_all → eventually crashed with
ExpiredTestDoubleError on the inherited InstanceDouble logger, and
concurrently raced ObjectSpace iteration with extractor_spec's
extract_all in the next spec, causing failure (2).

Fix:
- Bump both wait_for_idle calls to 30s, matching the other e2e test in
  the same file (line 84).
- Add `expect(...).to be true` to the first wait_for_idle, matching the
  pattern already used at the second wait_for_idle in the same test.
- Wrap the test body in begin/ensure so component.shutdown! always runs
  — a mid-test failure must not leak the scheduler thread and pollute
  subsequent specs in the same rspec process.

Verified on Ruby 2.6.10 locally:
  bundle exec rspec spec/datadog/symbol_database/component_spec.rb \
    spec/datadog/symbol_database/remote_config_integration_spec.rb \
    spec/datadog/symbol_database/extractor_spec.rb
  → 181 examples, 0 failures

Also verified on Ruby 3.2.3: 223 examples, 0 failures across the
related spec files.

Note: this is a fix to a test added in merged PR #5697. It's pulled
into this PR (#5871) because the bug surfaces from this PR's batch
composition (4 new tests shift the random seed). Independent of
#5871's other follow-up fixes.

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos core Involves Datadog core libraries debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants