Skip to content

SymDB: Component / Remote / Uploader integration#5717

Merged
p-datadog merged 24 commits into
masterfrom
symbol-database-upload-rebased
May 18, 2026
Merged

SymDB: Component / Remote / Uploader integration#5717
p-datadog merged 24 commits into
masterfrom
symbol-database-upload-rebased

Conversation

@p-datadog

@p-datadog p-datadog commented May 8, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Squashed rebase of #5431 (symbol-database-upload, 185 commits) onto current master, capturing only the unique work that hasn't been split out and merged via individual sub-PRs.

Adds the wiring layer that ties together the data models, extractor, batcher, uploader, and transport that were merged separately:

  • lib/datadog/symbol_database/component.rb — Component class for symbol_database lifecycle management
  • lib/datadog/symbol_database/remote.rb — Remote module for remote config integration
  • lib/datadog/symbol_database/extractor.rb — cache Module#name as an UnboundMethod constant (avoids resolving it per call in the hot extraction loop)
  • lib/datadog/symbol_database/scope.rb — minor comment edit on the targetable_lines Struct member
  • lib/datadog/core/configuration/components.rb, core/remote/client/capabilities.rb — top-level wiring to register symbol_database with the tracer and remote config
  • sig/datadog/symbol_database/component.rbs, remote.rbs — RBS declarations for the new files
  • sig/datadog/symbol_database/extractor.rbs, transport/http.rbs, uploader.rbs — RBS updates for the wrapped SymbolDatabase::Logger type and the cached MODULE_NAME constant
  • spec/datadog/symbol_database/component_initialize_spec.rb, component_spec.rb, integration_spec.rb, remote_config_integration_spec.rb, remote_spec.rb — coverage for the new component and RC integration
  • spec/datadog/symbol_database/extractor_spec.rb, scope_spec.rb, scope_batcher_spec.rb, service_version_spec.rb, uploader_spec.rb — secondary test updates from the rebase
  • spec/spec_helper.rb — platform-skip block (JRuby and Ruby < 2.6) for all symbol_database specs, with an opt-in symdb_supported_platforms tag for the negative tests that verify graceful degradation
  • docs/DynamicInstrumentation.md, docs/GettingStarted.md, docs/DevelopmentGuide.md — customer-facing and developer-facing documentation for the Symbol Database feature

Motivation:

#5431 has been the umbrella branch for the SymbolDatabase feature. Several pieces have been split out and merged independently (#5618, #5670, #5692, #5693, #5694). #5431 itself accumulated 185 commits and rebasing it directly produces conflicts on most commits because master now contains the post-squash final state of work that the branch held in intermediate form.

This PR captures the integration / wiring layer that hasn't been split out yet, on top of current master, in a single commit. #5431 is left intact for reference.

Change log entry

Yes. Dynamic Instrumentation: Symbol Database upload is now wired up — Ruby services will populate the Live Debugger UI's autocomplete (class names, method names, parameter names) when probes are being created.

How to test the change?

The new specs (component_spec.rb, integration_spec.rb, remote_config_integration_spec.rb, remote_spec.rb) exercise the wiring; CI runs them as part of the symbol_database test suite.

@p-datadog
p-datadog requested review from a team as code owners May 8, 2026 05:08
@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label May 8, 2026
@dd-octo-sts dd-octo-sts Bot added the core Involves Datadog core libraries label May 8, 2026
@dd-octo-sts

dd-octo-sts Bot commented May 8, 2026

Copy link
Copy Markdown
Contributor

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 16 steep:ignore comments, and clears 13 steep:ignore comments.

steep:ignore comments (+16-13)Introduced:
lib/datadog/symbol_database/component.rb:170
lib/datadog/symbol_database/extractor.rb:254
lib/datadog/symbol_database/extractor.rb:266
lib/datadog/symbol_database/extractor.rb:280
lib/datadog/symbol_database/extractor.rb:425
lib/datadog/symbol_database/extractor.rb:536
lib/datadog/symbol_database/extractor.rb:566
lib/datadog/symbol_database/extractor.rb:659
lib/datadog/symbol_database/extractor.rb:723
lib/datadog/symbol_database/extractor.rb:724
lib/datadog/symbol_database/extractor.rb:804
lib/datadog/symbol_database/extractor.rb:805
lib/datadog/symbol_database/extractor.rb:853
lib/datadog/symbol_database/remote.rb:106
lib/datadog/symbol_database/remote.rb:113
lib/datadog/symbol_database/scope.rb:87
Cleared:
lib/datadog/symbol_database/extractor.rb:245
lib/datadog/symbol_database/extractor.rb:257
lib/datadog/symbol_database/extractor.rb:271
lib/datadog/symbol_database/extractor.rb:416
lib/datadog/symbol_database/extractor.rb:527
lib/datadog/symbol_database/extractor.rb:557
lib/datadog/symbol_database/extractor.rb:650
lib/datadog/symbol_database/extractor.rb:714
lib/datadog/symbol_database/extractor.rb:715
lib/datadog/symbol_database/extractor.rb:795
lib/datadog/symbol_database/extractor.rb:796
lib/datadog/symbol_database/extractor.rb:844
lib/datadog/symbol_database/scope.rb:89

Untyped methods

This PR introduces 1 partially typed method. It increases the percentage of typed methods from 62.57% to 63.0% (+0.43%).

Partially typed methods (+1-0)Introduced:
sig/datadog/symbol_database/remote.rbs:22
└── def self.parse_config: (Datadog::Core::Remote::Configuration::Content content) -> ::Hash[::String, untyped]?

Untyped other declarations

This PR introduces 2 untyped other declarations, and clears 2 untyped other declarations. It increases the percentage of typed other declarations from 78.52% to 78.86% (+0.34%).

Untyped other declarations (+2-2)Introduced:
sig/datadog/symbol_database/extractor.rbs:11
└── @logger: untyped
sig/datadog/symbol_database/extractor.rbs:12
└── @settings: untyped
Cleared:
sig/datadog/symbol_database/extractor.rbs:10
└── @logger: untyped
sig/datadog/symbol_database/extractor.rbs:11
└── @settings: untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a83782a05

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/symbol_database/component.rb
@p-datadog
p-datadog marked this pull request as draft May 8, 2026 05:22
p-datadog pushed a commit that referenced this pull request May 8, 2026
Fix three issues identified in di-pr-review:

1. extractor_spec.rb:1809 — class fixture name mismatch.
   `class ExtractAllInjectableTest` was the pre-rename name; the
   test references `ExtractAllTargetableTest` post-rename (#5692).
   The mismatch was introduced by `git merge --squash -X theirs`,
   which favored the branch's pre-rename version on conflict while
   the surrounding references use master's renamed version. Tests
   were unrunnable as written. Renamed the fixture class to match.

2. integration_spec.rb:90, 115 — code comments still said "Injectable
   lines" while referencing `targetable_lines` / `targetable_lines?`
   methods. Updated comments.

3. component.rb:52, 56 — added YARD docs for the class methods
   `uploaded_this_process?` and `mark_uploaded` for consistency with
   the rest of the file's documentation style.

Not addressed (intentional):

- Sleep usage in component_spec.rb — used for negative-assertion
  tests ("verify nothing happens given X seconds"). Comments
  justify duration; the cleanest deterministic alternative would
  require adding a "scheduler decision made" hook to production
  code, which is scope expansion beyond this PR.

- Missing telemetry in component.rb / remote.rb rescue blocks —
  matches the established SymDB direction (#5693 explicitly
  removed telemetry from Extractor). Adding telemetry here would
  contradict the recent project decision.
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented May 8, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 96.65%
Overall Coverage: 97.14% (-0.04%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c46224b | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts dd-octo-sts Bot added the debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database) label May 8, 2026
@pr-commenter

pr-commenter Bot commented May 11, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-05-18 14:28:35

Comparing candidate commit c46224b in PR branch symbol-database-upload-rebased with baseline commit 21ea64b in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@p-datadog
p-datadog force-pushed the symbol-database-upload-rebased branch from 1a95405 to 4473fe6 Compare May 11, 2026 13:32
p-datadog pushed a commit that referenced this pull request May 11, 2026
Fix three issues identified in di-pr-review:

1. extractor_spec.rb:1809 — class fixture name mismatch.
   `class ExtractAllInjectableTest` was the pre-rename name; the
   test references `ExtractAllTargetableTest` post-rename (#5692).
   The mismatch was introduced by `git merge --squash -X theirs`,
   which favored the branch's pre-rename version on conflict while
   the surrounding references use master's renamed version. Tests
   were unrunnable as written. Renamed the fixture class to match.

2. integration_spec.rb:90, 115 — code comments still said "Injectable
   lines" while referencing `targetable_lines` / `targetable_lines?`
   methods. Updated comments.

3. component.rb:52, 56 — added YARD docs for the class methods
   `uploaded_this_process?` and `mark_uploaded` for consistency with
   the rest of the file's documentation style.

Not addressed (intentional):

- Sleep usage in component_spec.rb — used for negative-assertion
  tests ("verify nothing happens given X seconds"). Comments
  justify duration; the cleanest deterministic alternative would
  require adding a "scheduler decision made" hook to production
  code, which is scope expansion beyond this PR.

- Missing telemetry in component.rb / remote.rb rescue blocks —
  matches the established SymDB direction (#5693 explicitly
  removed telemetry from Extractor). Adding telemetry here would
  contradict the recent project decision.
p-datadog pushed a commit to p-datadog/dd-trace-rb that referenced this pull request May 11, 2026
Removes the `telemetry` parameter from `Datadog::SymbolDatabase::ScopeBatcher`
and the three `@telemetry&.report(...)` calls in its rescue blocks
(`add_scope`, `timer_loop`, `perform_upload`). The rescue blocks still log
at debug level.

Constructor change: `ScopeBatcher.new(uploader, logger:, telemetry: nil,
on_upload: nil, timer_enabled: true)` becomes `ScopeBatcher.new(uploader,
logger:, on_upload: nil, timer_enabled: true)`. Master has no non-test
callers of `ScopeBatcher.new` — only its spec instantiates it directly —
so this is contained to the tests in this PR.

The three "logs and reports telemetry without propagating" / "logs and
reports telemetry without crashing" rescue-path tests are rewritten to
assert on the log message only (the dropped telemetry assertion).

Motivation: matches the SymDB direction set by DataDog#5693 (telemetry removal
from Extractor). Telemetry reporting for internal SymDB lifecycle errors
is being replaced by debug-level logging; only the upload boundary
(Uploader) retains telemetry reporting.

Extracted from DataDog#5717.

Verification:
- bundle exec rspec spec/datadog/symbol_database/scope_batcher_spec.rb
  → 30 examples, 0 failures
- bundle exec steep check lib/datadog/symbol_database/scope_batcher.rb
  → No type error detected
- bundle exec standardrb lib/datadog/symbol_database/scope_batcher.rb
  spec/datadog/symbol_database/scope_batcher_spec.rb → clean

Co-Authored-By: Claude <[email protected]>
p-datadog pushed a commit to p-datadog/dd-trace-rb that referenced this pull request May 11, 2026
Adds a single telemetry call to `Datadog::SymbolDatabase::Uploader#upload_scopes`:

    @telemetry&.distribution('tracers', 'symbol_database.payload_size',
                             compressed_data.bytesize)

emitted unconditionally after GZIP compression and before the
MAX_PAYLOAD_SIZE check. The metric records the compressed payload size in
bytes for every upload attempt, including ones that get dropped for being
too large (so the oversized case is observable instead of silently lost).

Four new specs cover:
- successful upload: distribution called with the compressed bytesize
- oversized-skip path: distribution still called before the skip
- serialization-raises path: distribution not called
- compression-raises path: distribution not called

No constructor or API changes — the `telemetry:` parameter already exists
on Uploader.

Motivation: Uploader is the externally-observable boundary of the SymDB
subsystem. The payload_size distribution gives ops/customer visibility into
the size of uploaded symbol databases.

Extracted from DataDog#5717 to land independently.

Verification:
- bundle exec rspec spec/datadog/symbol_database/uploader_spec.rb
  → 21 examples, 0 failures
- bundle exec steep check lib/datadog/symbol_database/uploader.rb
  → No type error detected
- bundle exec standardrb lib/datadog/symbol_database/uploader.rb
  spec/datadog/symbol_database/uploader_spec.rb → clean

Co-Authored-By: Claude <[email protected]>
p-ddsign and others added 9 commits May 11, 2026 10:10
Squashed rebase of #5431 (symbol-database-upload, 185 commits) onto
current master, capturing only the unique work that hasn't been
split out and merged via individual sub-PRs (#5618, #5670, #5692,

Adds the wiring layer that ties together the data models, extractor,
batcher, uploader, and transport that were merged separately:
- Component class for symbol_database lifecycle management
- Remote module for remote config integration
- Component-initialize / integration / remote_config_integration tests
- Top-level wiring in core/configuration/components.rb and
  core/remote/client/capabilities.rb
- Settings registration in supported-configurations.json
- Documentation updates in DynamicInstrumentation.md and GettingStarted.md

Branch #5431 is left intact for reference; this is a fresh single-commit
PR for review.
Remove "yet" qualifiers from DynamicInstrumentation.md statements about
unsupported features (the qualifier implies an active deferral that
isn't being committed to). Remove cross-language gap bullets that name
internal mechanics (FIELD symbols, LOCAL scopes, fork deduplication,
injectable line information) without customer-facing impact, and the
class-methods note that described an internal extraction-vs-upload
detail. Rename "Deferred features" to "Limitations".

Co-Authored-By: Claude <[email protected]>
Fix three issues identified in di-pr-review:

1. extractor_spec.rb:1809 — class fixture name mismatch.
   `class ExtractAllInjectableTest` was the pre-rename name; the
   test references `ExtractAllTargetableTest` post-rename (#5692).
   The mismatch was introduced by `git merge --squash -X theirs`,
   which favored the branch's pre-rename version on conflict while
   the surrounding references use master's renamed version. Tests
   were unrunnable as written. Renamed the fixture class to match.

2. integration_spec.rb:90, 115 — code comments still said "Injectable
   lines" while referencing `targetable_lines` / `targetable_lines?`
   methods. Updated comments.

3. component.rb:52, 56 — added YARD docs for the class methods
   `uploaded_this_process?` and `mark_uploaded` for consistency with
   the rest of the file's documentation style.

Not addressed (intentional):

- Sleep usage in component_spec.rb — used for negative-assertion
  tests ("verify nothing happens given X seconds"). Comments
  justify duration; the cleanest deterministic alternative would
  require adding a "scheduler decision made" hook to production
  code, which is scope expansion beyond this PR.

- Missing telemetry in component.rb / remote.rb rescue blocks —
  matches the established SymDB direction (#5693 explicitly
  removed telemetry from Extractor). Adding telemetry here would
  contradict the recent project decision.
Previous commit eba8297 removed these bullets entirely; the user's
quote scoped removal to the parenthetical and explanation only.

Co-Authored-By: Claude <[email protected]>
Replace `untyped` with concrete types where Steep can verify:

component.rbs:
- @settings, @agent_settings → Core::Configuration::Settings / AgentSettings
- attr_reader settings → Core::Configuration::Settings
- self.build / initialize signatures → typed settings/agent_settings params

remote.rbs:
- capabilities return → Array[Integer] (capability flags)
- receivers telemetry → Core::Telemetry::Component
- receiver block params → Repository + ::Array[untyped]
- enable_upload / parse_config content → Core::Remote::Configuration::Content

@logger and the Repository::Change union remain `untyped`:

- @logger: callers pass Core::Logger but downstream collaborators
  (ScopeBatcher) expect SymbolDatabase::Logger, which has additional
  methods like #trace. Resolving the cascade is out of scope for a
  type-fillout commit. Matches the convention in lib/datadog/di/
  component.rbs.

- Change union (Inserted | Updated | Deleted): Steep cannot narrow
  `case change.type` to the corresponding variant, so concrete typing
  produces 13 false positives on `change.content` / `change.previous`.

Verified with `bundle exec steep check` — no errors.
…json

DD_INTERNAL_FORCE_SYMBOL_DATABASE_UPLOAD and DD_SYMBOL_DATABASE_UPLOAD_ENABLED
were already added in alphabetically-sorted positions by the earlier SymDB
configuration settings change. The squash rebase reintroduced them mid-file
(out of order, before DD_ENV). Remove the duplicates; the originals at the
sorted positions remain authoritative.

Verification:
- ruby -rjson -e 'JSON.parse(File.read("supported-configurations.json"))' → valid
- Each key appears once (line 415 and line 688)

Co-Authored-By: Claude <[email protected]>
The directory-based skip in spec_helper centralizes the platform guard for the
symbol_database suite (which requires MRI Ruby 2.6+). Expand the comment to
show how to opt a spec out of the wholesale skip via the
symdb_supported_platforms metadata tag, including describe-level and
example-level usage examples — needed when a spec validates the platform
guard itself and therefore must run on the otherwise-skipped platforms.

Verification:
- ruby -c spec/spec_helper.rb → syntax OK
- Comment-only change; no behavior modified.

Co-Authored-By: Claude <[email protected]>
The squash rebase replaced per-line @type var annotations with three
steep:ignore:start/:end blocks in scope_batcher.rb, and dropped :: prefixes
from concrete-type references in scope_batcher.rbs. Both changes inflated
the diff against the merged ScopeBatcher PR's form without changing
behavior.

Restore the original style:
- add_scope/flush/shutdown: `# @type var scopes_to_upload: ::Array[Scope]?`
  re-added on each local declaration; the surrounding steep:ignore blocks
  are removed (the annotation does the same job in two lines instead of six).
- shutdown's guard widened to `unless scopes_to_upload.nil? || .empty?` so
  Steep accepts `.empty?` on the now-nullable-annotated local.
- scope_batcher.rbs: `::` restored on MAX_SCOPES, INACTIVITY_TIMEOUT,
  MAX_FILES, @on_upload, @scopes, @Mutex, @timer_cv, @timer_thread,
  @file_count, @uploaded_modules, perform_upload arg, size return.

Verification:
- bundle exec steep check lib/datadog/symbol_database/scope_batcher.rb
  → No type error detected.
- bundle exec rspec spec/datadog/symbol_database/scope_batcher_spec.rb
  → 26 examples, 0 failures.

Co-Authored-By: Claude <[email protected]>
Six small changes flagged in review on the Component class:

- build: `unless environment_supported?(symdb_logger) ... return nil end`
  collapsed to guard-clause `return nil unless environment_supported?(...)`,
  with a note that the helper logs the specific reason internally.
- build: compound `unless settings.remote&.enabled || ... force_upload`
  flipped to `if !A && !B` (DeMorgan) — easier to read.
- shutdown!: removed standalone `# @api private` marker before `private`
  keyword (redundant; the keyword already enforces visibility).
- scheduler_loop: added a comment explaining what `should_fire = true`
  means (debounce deadline elapsed; extract_and_upload runs once after
  the mutex is released).
- scheduler_loop: removed the steep:ignore:start/:end block around the
  mutex section. NOTE: removal is unverified locally because the libdatadog
  version mismatch blocks Steep from running here; CI will surface a
  regression if removal isn't safe.
- extract_and_upload: moved `extraction_duration` and `targetable_count`
  calculations inside the `@logger.debug` block so the work is skipped
  when debug logging is off (the latter iterates the scope tree).

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog force-pushed the symbol-database-upload-rebased branch from dda2762 to 05794a2 Compare May 11, 2026 14:12
@p-datadog p-datadog changed the title SymDB: Component / Remote / Uploader integration (squashed rebase of #5431) SymDB: Component / Remote / Uploader integration May 11, 2026
p-ddsign and others added 2 commits May 11, 2026 10:20
Replaces remaining `untyped` declarations introduced by this PR with
concrete types where the type is unambiguous, and removes phantom sig
declarations that referenced methods/fields not present in the production
code.

- symbol_database.rbs: remove phantom `@mutex`, `@component`,
  `self.component`, `self.set_component`, `self.enabled?` — none exist
  in `lib/datadog/symbol_database.rb`. The sig had been declaring an API
  that never landed.
- component.rbs: `@logger`, `initialize`'s `logger` parameter, and
  `environment_supported?`'s `logger` parameter typed as
  `SymbolDatabase::Logger` (the wrapper instantiated inside `build`).
  `build`'s `logger` parameter stays `untyped` — it's the raw core
  logger passed in from the configuration layer (duck-typed, matches
  existing `untyped` convention for cross-component logger handoff).
- remote.rbs: `process_change`'s `change` parameter and `receiver`
  block's `changes` parameter typed as
  `Datadog::Core::Remote::Configuration::Repository::change` (a type
  alias defined in repository.rbs as
  `Change::Deleted | Change::Inserted | Change::Updated`). The
  `parse_config` return value's inner `untyped` stays — the JSON-decoded
  hash is genuinely heterogeneous.

Verification:
- ruby -rrbs parser run on all three files: OK
- bundle exec steep check could not run locally (libdatadog version
  mismatch blocks the dev env here); CI will surface any regression.

Co-Authored-By: Claude <[email protected]>
CI flagged 10 Steep errors after the previous type-fill commit. Three
distinct issues:

1. Component @logger (typed as SymbolDatabase::Logger) passed to Uploader
   whose sig said `Datadog::Core::Logger`. Runtime reality: the wrapped
   Logger is what's passed (and forwarded through to Transport::HTTP).
   Fix: update Uploader and Transport::HTTP sigs to declare
   `SymbolDatabase::Logger` — matches the actual call path.

2. Remote.process_change's `change` parameter typed as the union
   `Change::Deleted | Change::Inserted | Change::Updated` — Steep cannot
   narrow the union via the `case change.type` symbol-dispatch the code
   uses. Add `# @type var change: <ConcreteClass>` annotations inside
   each `when` branch — narrows for Steep without changing the dispatch
   (preserves test patterns using `instance_double('Change', type: ...)`).
   The `else` and rescue branches use respond_to? introspection which
   Steep cannot narrow either — silenced with per-line steep:ignore.

3. scheduler_loop's `@scheduled_at - Time.get_time` flagged because
   Steep does not narrow nullable instance variables across an
   `if @scheduled_at.nil?` check. Copy to local variable so Steep
   narrows in the else branch.

Verification:
- bundle exec steep check → No type error detected.
- bundle exec rspec spec/datadog/symbol_database/{remote,component,uploader}_spec.rb
  → 71 examples, 0 failures.

Co-Authored-By: Claude <[email protected]>
p-datadog pushed a commit that referenced this pull request May 11, 2026
* origin/base-5733:
  Add Rails integration test for symbol database extraction
  SymDB: add extraction performance benchmark
  Add Component#after_fork! for post-fork mutex hygiene + scheduler reset
  Fix Steep failures with type annotations, not untyped reverts
  Add TracePoint :class hot-load hook for incremental class extraction
  Fix Steep failures from over-tight type-fill + reinstate scheduler_loop ignore
  Fill in concrete types for remaining SymDB sig declarations
  Address review findings on component.rb
  Restore @type var annotations and :: prefixes in ScopeBatcher
  Document symbol_database platform-skip opt-out usage in spec_helper
  Remove duplicate SymDB env var entries from supported-configurations.json
  Fill in concrete types for SymDB Component and Remote sig files
  Restore Instance/Local variable extraction bullet lead-ins
  Address review findings on #5717
  SymDB docs: trim DI documentation cross-language disclaimers
  SymDB: Add Component, Remote, Uploader integration on top of master
Unicorn Enterprises and others added 2 commits May 13, 2026 19:35
`DD_INTERNAL_FORCE_SYMBOL_DATABASE_UPLOAD` is an internal testing knob, not
a customer-facing setting. Moves the snippet out of DynamicInstrumentation.md
(customer guide) into DevelopmentGuide.md (contributor guide) under a new
"Testing the Symbol Database without Remote Configuration" subsection.
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f0c6427b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/symbol_database/remote.rb Outdated
p-ddsign and others added 7 commits May 13, 2026 20:01
Three CRITICAL items from PR review:

1. Component built without telemetry — fixed by threading telemetry from
   Components#initialize → Component.build → Component.new → Uploader.new.
   Uploader's `telemetry: nil` default is now provided by the wiring layer.

2. Rescue blocks missing telemetry — added `@telemetry&.report` at:
   - component.rb start_upload, scheduler_loop, extract_and_upload
   - remote.rb component-lookup-in-receiver, process_change
   Telemetry in the Remote module is looked up once per receiver invocation
   via Datadog.send(:components)&.telemetry (component-boundary exception).
   parse_config no longer rescues JSON::ParserError internally — the error
   propagates to process_change's rescue which logs + reports telemetry.

3. Sleep in tests — removed all 8 sleeps. Analysis shows they were guarding
   against scheduler-thread races that the production code already handles
   deterministically:
   - start_upload short-circuits on uploaded_this_process?/@shutdown before
     starting a scheduler thread — assert @scheduler_thread.nil? instead
   - shutdown! signals the CV and joins the thread before returning — the
     thread cannot fire after shutdown! returns
   - the `after { component.shutdown! }` hook joins any live scheduler thread
     before RSpec verifies `not_to receive` expectations

318/318 symdb specs pass; Steep clean on changed files.
…guard

Address review comment: the receiver block passed to
`Core::Remote::Dispatcher::Receiver.new` is stored as a Proc and invoked later
via `@block.call(...)`. Using `return` inside that block raises `LocalJumpError:
unexpected return` once the enclosing `receivers` method has returned. This
fires whenever `Datadog.send(:components).symbol_database` is nil — i.e. on
JRuby, Ruby 2.5, when symbol_database is disabled, or when remote config is
disabled without force_upload — and kills RC processing for that poll instead
of just no-op'ing the SymDB config.

- Fixed in lib/datadog/symbol_database/remote.rb:53 by wrapping the change loop
  in `if component ... end` (matches the existing DI receiver pattern at
  lib/datadog/di/remote.rb:41). The `# steep:ignore ReturnTypeMismatch`
  annotation is removed since it was only there because of the early return.
- Test added in spec/datadog/symbol_database/remote_spec.rb that invokes the
  stored receiver block with a nil symbol_database component and asserts no
  LocalJumpError, plus a companion test for the present-component path.

Verified: `bundle exec rspec spec/datadog/symbol_database/remote_spec.rb` (18
examples, 0 failures); `bundle exec steep check lib/datadog/symbol_database/remote.rb`
clean.

Co-Authored-By: Claude <[email protected]>
`Datadog.send(:components)` defaults to `allow_initialization: true`, which
synchronously builds the entire component tree if not yet built. The RC receiver
callback should never trigger initialization — it is a state lookup, not a
constructor, and triggering startup from inside an RC dispatch thread has the
wrong ownership semantics.

Pass `allow_initialization: false` at both call sites (the component lookup in
the receiver block, and `lookup_telemetry`). When components do not exist the
lookups return nil and the receiver no-ops via the `if component` guard added
in the previous commit. This matches the established pattern in
core/telemetry/logger.rb, kit/appsec/events/v2.rb, core/crashtracking/component.rb,
profiling/ext/exec_monkey_patch.rb, and tracing.rb.

Tests in spec/datadog/symbol_database/remote_spec.rb updated to match the new
call signature.

Verified: `bundle exec rspec spec/datadog/symbol_database/remote_spec.rb` (18
examples, 0 failures); `bundle exec steep check lib/datadog/symbol_database/remote.rb`
clean.

Co-Authored-By: Claude <[email protected]>
`@file_count += 1` was moved from after the dedup check to before it during
the squash rebase of PR #5431 (commit 3eca473). This silently reverted
the fix from PR #5431 commit 0f60682 ("Fix file-count to only track
unique accepted scopes") that addressed a codex review finding. The fix
was carried into master via PR #5691 — the rebase regressed it.

With duplicates consuming the MAX_FILES budget, a re-extraction scenario
(e.g. Rails reload re-offering the same scopes) exhausts the unique-file
budget after MAX_FILES duplicate offerings, rejecting subsequent unique
scopes that should still be accepted.

- Fixed in lib/datadog/symbol_database/scope_batcher.rb:75-91 (add_scope)
  by moving @file_count += 1 to after the dedup check, matching master.
- Restored the explanatory comments on the two checks describing why
  duplicates do not count toward MAX_FILES.
- Restored the regression test 'does not count duplicates toward the
  file limit' in spec/datadog/symbol_database/scope_batcher_spec.rb (the
  test that previously caught this regression and was deleted alongside
  the bug's reintroduction).

Verified: bundle exec rspec spec/datadog/symbol_database/scope_batcher_spec.rb
(30 examples, 0 failures).

Co-Authored-By: Claude <[email protected]>
The squash rebase of PR #5431 (commit 3eca473) replaced named-constant
references with magic numbers and dropped two assertions from the
'continues batching after upload' test in scope_batcher_spec.rb. The
constant MAX_SCOPES = 400 is still defined and documented (value matches
Java and Python tracers for cross-language consistency), so the
substitutions were pure regressions, not deliberate cleanup.

- spec/datadog/symbol_database/scope_batcher_spec.rb:52 — restored
  expect(scopes.size).to eq(described_class::MAX_SCOPES)
- :55 — restored described_class::MAX_SCOPES.times do |i|
- :69 — restored (described_class::MAX_SCOPES + 1).times do |i|
- :64-76 — restored the upload_calls capture and the two dropped
  assertions: expect(upload_calls.size).to eq(1) and
  expect(upload_calls.first.size).to eq(described_class::MAX_SCOPES).
  Without these, the test no longer verifies that an upload happened
  with exactly MAX_SCOPES scopes — it only checked the leftover-scope
  count.

Verified: bundle exec rspec spec/datadog/symbol_database/scope_batcher_spec.rb
(30 examples, 0 failures).

Co-Authored-By: Claude <[email protected]>
Three closely-related regressions from the squash rebase of PR #5431
(commit 3eca473), all in the shutdown/reset region:

1. TIMER_JOIN_TIMEOUT constant: was extracted to a named constant with
   a doc comment explaining the bound; the rebase inlined it back to a
   bare 5 with a stale trailing comment.

2. thread_to_join capture-under-mutex pattern (in both shutdown and
   reset): the documented defensive pattern was removed. Without it, a
   concurrent add_scope can create a new timer thread between the
   `@timer_thread = nil` assignment and the `&.join(...)` call,
   orphaning the original thread (the join hits nil or the new thread).
   The original code captured the thread reference under the mutex,
   nil'd the field under the mutex, then joined the captured reference
   outside the mutex.

3. reset visibility: was private with the doc comment "Private so
   production code cannot accidentally invoke it; tests call via
   send(:reset)". The rebase moved it to public and replaced the
   comment with an @api private annotation that contradicted the
   actual visibility. Restored to private; updated spec to use
   context.send(:reset).

Also restored TIMER_JOIN_TIMEOUT declaration to the RBS sig file and
moved reset from the public section to the private section to match.

Verified: bundle exec rspec spec/datadog/symbol_database/scope_batcher_spec.rb
(30 examples, 0 failures); bundle exec steep check
lib/datadog/symbol_database/scope_batcher.rb (no errors).

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog marked this pull request as ready for review May 14, 2026 20:36

@janine-c janine-c left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Some optional writing suggestions to bring the .md files a little more in line with the docs style guide 🙂

Comment thread docs/GettingStarted.md Outdated
Comment thread docs/GettingStarted.md Outdated
Comment thread docs/DevelopmentGuide.md Outdated
Comment thread docs/DynamicInstrumentation.md Outdated
Comment thread lib/datadog/core/configuration/supported_configurations.rb Outdated
Comment thread lib/datadog/symbol_database/component.rb
Comment thread lib/datadog/symbol_database/component.rb Outdated
Comment thread lib/datadog/symbol_database/remote.rb Outdated
Comment thread sig/datadog/symbol_database/component.rbs Outdated
Comment thread sig/datadog/symbol_database/component.rbs Outdated
Comment thread sig/datadog/symbol_database/component.rbs Outdated
Comment thread sig/datadog/symbol_database/component.rbs Outdated
Comment thread sig/datadog/symbol_database/extractor.rbs Outdated
Comment thread sig/datadog/symbol_database/remote.rbs Outdated
p-ddsign added 4 commits May 18, 2026 09:59
Address review suggestions on Symbol Database doc paragraphs:

- docs/GettingStarted.md: use "for example" instead of "e.g." in the
  DD_DYNAMIC_INSTRUMENTATION_REDACTED_TYPES row
- docs/GettingStarted.md: rephrase the Symbol Database intro paragraph
  ("activates automatically using Remote Configuration")
- docs/DevelopmentGuide.md: rephrase the "Testing the Symbol Database
  without Remote Configuration" intro for clearer flow
- docs/DynamicInstrumentation.md: rephrase the Enabling section to match
  the same "using Remote Configuration when you open the DI UI" wording
Both DD_INTERNAL_FORCE_SYMBOL_DATABASE_UPLOAD (line 71) and
DD_SYMBOL_DATABASE_UPLOAD_ENABLED (line 109) were already registered in
supported_configurations.rb. Earlier additions in this branch added them
again, producing duplicates. Drop the duplicates so each env var is
registered exactly once.
Two small fixes from review:

1. Component#shutdown! previously set @shutdown=true under both
   @scheduler_mutex and @Mutex, and Component#shutdown? read it under
   @Mutex. The dual-mutex pattern is confusing and the @Mutex set was
   redundant — start_upload and scheduler_loop both read @shutdown under
   @scheduler_mutex, which is the primary guard. Move shutdown? to read
   under @scheduler_mutex and drop the redundant @shutdown=true in the
   @Mutex block of shutdown! (which now only waits for any in-flight
   extraction to finish).

   The advisory race between an external shutdown? check and a subsequent
   start_upload is closed by start_upload's own @scheduler_mutex re-check —
   no extraction can begin after shutdown.

2. SymbolDatabase::Remote::PRODUCT was defined inside `class << self`,
   which placed it on the singleton class instead of the module. Move it
   to module level so external code (and the rbs declaration) can
   reference it as `Datadog::SymbolDatabase::Remote::PRODUCT`.
Apply review suggestions across the symbol_database RBS files: prefix
Ruby Core types (Thread::Mutex, Thread::ConditionVariable, Time, Integer,
Float, Thread, UnboundMethod, Logger, Numeric, Array, Hash, String) with
`::` so they unambiguously resolve to the top-level types instead of
being looked up under Datadog::SymbolDatabase. Matches the convention
used in other dd-trace-rb RBS files.

- component.rbs: class-level and instance ivars, build's logger param,
  attr_reader return types, wait_for_idle timeout, log_scope_tree depth,
  count_targetable_methods arg/return
- extractor.rbs: MODULE_SINGLETON_CLASS_PRED and MODULE_NAME UnboundMethod
- remote.rbs: PRODUCT, products, capabilities, receivers, receiver,
  parse_config — all String/Array/Hash/Integer references; also expand
  Dispatcher::Receiver / Configuration::Repository::change to fully
  qualified paths
@p-datadog

Copy link
Copy Markdown
Member Author

Thanks for the docs polish — all four inline suggestions applied verbatim in 1f95b81.

@p-datadog
p-datadog merged commit 87c5373 into master May 18, 2026
584 checks passed
@p-datadog
p-datadog deleted the symbol-database-upload-rebased branch May 18, 2026 15:59
@dd-octo-sts dd-octo-sts Bot added this to the 2.34.0 milestone May 18, 2026
p-datadog added a commit that referenced this pull request May 18, 2026
@Strech Strech mentioned this pull request May 27, 2026
p-datadog pushed a commit that referenced this pull request May 27, 2026
- Add the "opt-in" qualifier to the #5717 Symbol Database line to match
  the v2.34.0 release notes verbatim
- Remove the "### Changed" section (#5776) that does not appear in the
  v2.34.0 release announcement; drop the now-unused [#5776] link

Co-Authored-By: Claude Opus 4.7 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos core Involves Datadog core libraries debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants