Skip to content

DI: backfill CodeTracker registry with iseqs for pre-loaded files#5496

Merged
p-datadog merged 46 commits into
masterfrom
di-iseq-backfill
Apr 22, 2026
Merged

DI: backfill CodeTracker registry with iseqs for pre-loaded files#5496
p-datadog merged 46 commits into
masterfrom
di-iseq-backfill

Conversation

@p-datadog

@p-datadog p-datadog commented Mar 23, 2026

Copy link
Copy Markdown
Member

What does this PR do?

When CodeTracker starts, backfills the iseq registry with instruction sequences for files that were loaded before tracking began. Also adds a DI.iseq_type C extension (Ruby 3.1+) for precise whole-file iseq detection.

Motivation:

CodeTracker only captures files loaded after DI starts via :script_compiled TracePoint events. Most application code and all gems are loaded at boot time before DI activates, making them invisible to line probes. This change walks the Ruby object space via the all_iseqs C extension (#5111) and recovers whole-file iseqs for already-loaded code.

Implementation:

  • CodeTracker#backfill_registry walks the object space via DI.file_iseqs, filters to whole-file iseqs, and stores them in the registry without overwriting entries from :script_compiled.
  • Whole-file iseq detection uses DI.iseq_type on Ruby 3.1+ (returns :top for require/load, :main for the entry script). On Ruby < 3.1, falls back to first_lineno == 0 — which produces the same result because Ruby's rb_iseq_new_top and rb_iseq_new_main pass INT2FIX(0) while method/class/block definitions get their source line (>= 1).
  • DI.iseq_type wraps the internal rb_iseq_type() function added in Ruby 3.1 (commit 89a02d89 by Koichi Sasada). Guarded behind have_func('rb_iseq_type') in extconf.rb — only compiled when the symbol exists.
  • Backfill runs inside start after the trace point is enabled, so files loaded concurrently are captured by the trace point (backfill won't overwrite them).
  • compile_file iseqs are excluded via first_lineno == 0 check on all Ruby versions. Targeted TracePoints are bound to the specific iseq object — a compile_file iseq is distinct from the require-produced iseq, so a probe installed against it silently never fires. Verified by reading vm_trace.c and iseq.c.

Change log entry

Yes. DI/LD: more third-party libraries are now instrumentable in DI/LD.

Additional Notes:

Documentation: Updated in #5501. While this PR qualitatively improves which third-party code is instrumentable, in our testing the gains from whole-file backfill alone are ~10% of pre-loaded files (the rest have GC'd top iseqs and require per-method backfill from #5501). To avoid churn, #5501 removes the entire "Code Tracking Requirement" section from docs/DynamicInstrumentation.md; this PR intentionally keeps the existing docs as-is.

How to test the change?

Unit tests for backfill_registry (filtering, overwrite protection, error boundary, fallback path, compile_file exclusion). Integration tests: backfill path (load before tracking → probe installs and fires → snapshot captured) and :script_compiled path (load after tracking → probe installs and fires → snapshot captured). Bootsnap integration test (requires bootsnap gem, runs in di:bootsnap CI job). TracePoint binding tests verifying compile_file iseqs don't fire for require-produced code.

Manually tested in gobo by inspecting the DI code tracer registry. Before:

2026-03-24_19-29-01

After:

2026-03-24_19-29-24 2026-03-24_19-29-33

Manual testing via gobo targeting stdlib:

2026-03-24_20-28-16 2026-03-24_20-28-25

@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Mar 23, 2026
@github-actions github-actions Bot added the profiling Involves Datadog profiling label Mar 23, 2026
@github-actions

github-actions Bot commented Mar 23, 2026

Copy link
Copy Markdown

Typing analysis

Note: Ignored files are excluded from the next sections.

Untyped methods

This PR introduces 1 partially typed method, and clears 1 partially typed method. It increases the percentage of typed methods from 61.29% to 61.33% (+0.04%).

Partially typed methods (+1-1)Introduced:
sig/datadog/di/code_tracker.rbs:15
└── def iseqs_for_path_suffix: (String suffix) -> untyped
Cleared:
sig/datadog/di/code_tracker.rbs:14
└── def iseqs_for_path_suffix: (String suffix) -> untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Mar 24, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 84.96%
Overall Coverage: 97.20% (-0.07%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f36418f | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Mar 24, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-04-22 12:47:10

Comparing candidate commit f36418f in PR branch di-iseq-backfill with baseline commit 151a0e0 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd1e2d3b72

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/di/code_tracker.rb Outdated
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a104ffd302

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/di/code_tracker.rb
@p-datadog
p-datadog force-pushed the di-iseq-backfill branch 3 times, most recently from bd63696 to 45a9f83 Compare April 9, 2026 18:20
p-ddsign and others added 6 commits April 21, 2026 08:10
When CodeTracker starts, use the all_iseqs C extension to populate the
registry with instruction sequences for files that were loaded before
tracking began. This enables line probes on third-party code and
application code loaded at boot time.

Only whole-file iseqs (first_lineno == 0) are backfilled — per-method
iseqs require instrumenter changes to select the correct iseq for a
target line and will be supported in a follow-up.

Backfill does not overwrite entries from :script_compiled, which are
authoritative. The C extension availability is checked via
DI.respond_to?(:all_iseqs) so the code gracefully degrades when
the extension is not compiled.

- Added CodeTracker#backfill_registry
- Called from CodeTracker#start after trace point is enabled
- Added RBS signature
- Added tests for backfill behavior and C extension fallback

Co-Authored-By: Claude <[email protected]>
- Added rescue block around backfill_registry so failures are
  best-effort (logged + telemetry) rather than propagating
- Replaced all skip-based tests with mock-based tests that exercise
  backfill logic without requiring the compiled C extension
- Added tests for: mixed iseq types, multiple files, error handling,
  suffix/exact lookup on backfilled entries, start ordering
- 27 examples, 0 failures, 0 pending, 0 skipped

Co-Authored-By: Claude <[email protected]>
Tests the end-to-end flow: test class loaded before code tracking
starts → CodeTracker#start triggers backfill via all_iseqs C extension
→ iseq recovered from object space → line probe installed on
backfilled iseq → probe fires and captures local variables.

Runs under rake spec:di_with_ext (requires compiled C extension).

Three test cases:
- Probe installs successfully on backfilled iseq
- Probe fires when target line executes
- Snapshot captures local variables from backfilled iseq

Co-Authored-By: Claude <[email protected]>
On macOS CI the C extension is compiled, so backfill_registry populates
the CodeTracker registry with pre-loaded files during start. This broke
existing tests that expect the registry to be empty after start or to
contain exactly N explicitly-loaded files.

Fix by stubbing backfill_registry in test contexts that exercise
:script_compiled behavior. Backfill is tested separately in its own
describe blocks.

Affected contexts:
- CodeTracker #start (before block)
- CodeTracker shared context 'when code tracker is running'
- CodeTracker #iseqs_for_path_suffix (around block)
- Instrumenter shared context 'with code tracking'

Co-Authored-By: Claude <[email protected]>
…kfill

The backfill filter used first_lineno == 0 to identify whole-file iseqs,
but most whole-file iseqs from all_iseqs have first_lineno == 1. The new
DI.iseq_type method reads the iseq type directly from the Ruby VM struct
and returns a symbol (:top, :method, :block, :class, etc.).

The backfill now filters by type == :top || type == :main, which
correctly identifies whole-file iseqs regardless of first_lineno.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
rb_iseq_type is an internal Ruby function that only exists in Ruby 3.1+.
On Ruby 2.7 and 3.0, referencing it causes an undefined symbol error at
load time, crashing the entire C extension (including all_iseqs and
exception_message which work fine on those versions).

Use have_func in extconf.rb to detect rb_iseq_type at compile time, and
wrap the iseq_type function + registration in #ifdef HAVE_RB_ISEQ_TYPE.
The Ruby code in code_tracker.rb already handles the missing method via
DI.respond_to?(:iseq_type) with a first_lineno fallback.

Co-Authored-By: Claude <[email protected]>
dd-octo-sts Bot and others added 19 commits April 21, 2026 08:10
- Remove load_iseq from prepended module in after block instead of
  leaving the prepend as a permanent global mutation. Uses the same
  pattern as the DI instrumenter for method probe cleanup: the module
  stays in the ancestor chain but becomes a no-op.
- Regenerate ruby_3.2_rails8_mysql2.gemfile to include bootsnap.
  Other Ruby version gemfiles will be regenerated in their respective
  CI environments.

Co-Authored-By: Claude <[email protected]>
- Rewrite `unless` with mixed logical operators to `if` with negated
  conditions in backfill_registry (Style/UnlessLogicalOperators)
- Remove redundant begin block and line continuations in Bootsnap test

Co-Authored-By: Claude <[email protected]>
…oads

The skip guard only caught LoadError, but bootsnap can load successfully
while its iseq cache fails to initialize (e.g. C extension not compiled
for this environment). This caused test failures in CI jobs where
bootsnap was available as a transitive dependency but couldn't install
its iseq cache.

Fix: probe-test the iseq cache in before(:all) — install it in a temp
dir, clean up, and skip if any error occurs.

Co-Authored-By: Claude <[email protected]>
The require was inside before(:all), which loads bootsnap into the
process at test execution time — after other test infrastructure has
initialized. Moving it to file load time makes the side effect visible
and predictable. If bootsnap isn't available or its iseq cache can't
initialize, the constant BOOTSNAP_AVAILABLE is false and the entire
describe block skips via before(:all).

Also eliminates the standard:disable comment for Style/RedundantBegin
since the rescue is now in a top-level begin/rescue/end, not a block.

Co-Authored-By: Claude <[email protected]>
… RBS type

- compile_file_backfill_spec.rb: fix skip message "Ruby < 3.1" → "Ruby >= 3.1 only"
- iseq_type_spec.rb: rename "eval with top-level code" → "compiled top-level code"
  (the test uses compile(), not eval)
- code_tracker_spec.rb: remove two comments that just restate the expectation below them
- code_tracker_spec.rb: rename "calls backfill_registry after trace point is enabled"
  test — it didn't actually verify ordering, only that active? is true after start
- sig/datadog/di.rbs: iseq_type return type Symbol → Symbol? (C code returns Qnil
  when rb_iseqw_to_iseq returns NULL)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
The guard fires in zero CI jobs — the di:bootsnap job always has bootsnap
in the gemfile. It only fires when a developer runs the file manually without
the correct gemfile, in which case a LoadError is the right signal.

More importantly, the StandardError rescue around install!() could silently
skip all tests if bootsnap's C extension fails on a new Ruby version, hiding
a real compatibility break as green CI.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
The test "does not register compile_file iseqs" passed vacuously when
the code was correct — the `if result` block meant zero assertions ran
when result was nil. Add `expect(result).to be_nil` to directly assert
the intended behavior.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
require "bootsnap" alone does not load Bootsnap::CompileCache::ISeq —
that submodule must be required explicitly. Without it, the spec fails
with NameError when referencing Bootsnap::CompileCache::ISeq.

Also restores the BOOTSNAP_AVAILABLE skip guard that was removed, so
the tests degrade gracefully when bootsnap or its ISeq cache is
unavailable.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
The di:bootsnap CI task runs with rails8-mysql2 gemfile but does not
compile the DI C extension (that is done by di_with_ext). When
DI::Component.build calls environment_supported?, it finds
DI.respond_to?(:exception_message) is false and calls logger.warn().
The strict instance_double rejects the unexpected warn call, failing
the test before it can exercise any Bootsnap behavior.

Add a skip guard parallel to the existing BOOTSNAP_AVAILABLE guard.

Co-Authored-By: Claude <[email protected]>
- Add @return [void] YARD tag to backfill_registry
- Restructure compile_file_backfill_spec into two contexts:
  - "with iseq_type (Ruby 3.1+)" — existing test, skips on < 3.1
  - "with first_lineno fallback (all Ruby versions)" — new test that
    stubs iseq_type away and verifies the first_lineno == 0 heuristic
    also excludes compile_file iseqs (which have first_lineno == 1)
- Extract shared setup into helper method to avoid duplication

Co-Authored-By: Claude <[email protected]>
The previous fix added a skip guard for the DI C extension, which is a
build dependency not a platform limitation. That guard silently skipped
all bootsnap tests on every CI run.

Correct fix: add the compile dependency to the di:bootsnap Rake task,
matching the pattern used by di:di_with_ext. Remove the skip guard.

Co-Authored-By: Claude <[email protected]>
The di:bootsnap CI job runs against rails8-mysql2 gemfiles which include
bootsnap. The skip guard fires in zero CI jobs — it only fires when a
developer runs the file manually without bootsnap installed. That's a
local dev convenience skip, not a platform limitation.

If bootsnap can't load, the test should fail — that tells the developer
they need the right gemfile. Requires and initialization now happen at
file load time without rescue, so failures are immediate and visible.

Co-Authored-By: Claude <[email protected]>
The file-level Bootsnap::CompileCache::ISeq.install!(BOOTSNAP_VERIFY_DIR)
set Bootsnap's internal cache directory, which was then deleted.
Subsequent install!(cache_dir) calls in the before block may not update
the internal state, causing cache files to be written to the deleted
directory. Other bootsnap tests passed because they don't check for
cache files on disk.

Fix: remove the file-level install!/cleanup entirely. The requires
at file level are sufficient to prove bootsnap loads. The test's
before block handles install! with the correct cache directory.

Co-Authored-By: Claude <[email protected]>
Bootsnap's load_iseq intentionally returns nil when Coverage is active
(line 53 of bootsnap/compile_cache/iseq.rb), disabling iseq disk
caching entirely. This means no cache files are written during CI runs
with SimpleCov, causing the "Bootsnap cache was actually used" test to
fail — it checks for cache files on disk.

This is a genuine platform condition: Bootsnap + Coverage are
incompatible for disk caching. Other bootsnap tests (iseq capture,
TracePoint firing, probe installation, snapshot capture) pass because
they test the :script_compiled path, not the disk cache.

Root cause of the previous incorrect fix (removing file-level install!):
the file-level install! was not the problem — Coverage was.

Co-Authored-By: Claude <[email protected]>
This reverts commit 45a9f83281924b2b4321982b66bf619835ce7abf.
Bootsnap's load_iseq intentionally returns nil when Coverage is running
(SimpleCov in CI), disabling iseq disk caching entirely. This caused
the "Bootsnap cache was actually used" test to fail — no cache files
were written during the priming load.

Fix: suspend Coverage for the duration of each bootsnap test example.
Coverage.suspend (Ruby 3.2+) pauses coverage collection, allowing
Bootsnap's load_iseq to proceed normally. di:bootsnap runs on 3.2+
only, so Coverage.suspend is always available.

The suspend/resume is tracked via @coverage_was_suspended to avoid
calling Coverage.resume in local dev where Coverage was never started
(which would raise RuntimeError).

Co-Authored-By: Claude <[email protected]>
Two bugs caused "Bootsnap cache was actually used" to always fail:

1. ISeq.install!(cache_dir) sets cache_dir to "#{cache_dir}-iseq"
   internally but does not create the directory. Bootsnap's native
   fetch silently skips writing when the dir doesn't exist. Fix:
   mkdir_p the actual iseq cache dir after install!.

2. Dir.glob checked cache_dir but Bootsnap writes to cache_dir + "-iseq".
   Fix: glob the actual iseq cache dir from ISeq.cache_dir.

Also fix the after block: remove_method :load_iseq from the prepended
mixin is permanent — prepend is idempotent, so a subsequent install!
won't re-add load_iseq, breaking later examples. Instead, point the
cache dir at the deleted directory so load_iseq becomes a no-op.

The Coverage.suspend added in the previous commit is still needed:
Bootsnap's load_iseq returns nil when Coverage.running? is true,
bypassing the cache path entirely.

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog marked this pull request as ready for review April 21, 2026 12:13
@p-datadog
p-datadog requested review from a team as code owners April 21, 2026 12:13

@vpellan vpellan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left three non blocking comments

Comment thread lib/datadog/di/base.rb Outdated
Comment thread lib/datadog/di/base.rb Outdated
Comment thread Matrixfile Outdated
p-datadog and others added 2 commits April 22, 2026 08:10
Co-authored-by: Victor Pellan <[email protected]>
Address review comment: vpellan suggested initializing to [] to
simplify the type and remove the steep:ignore workaround.

- @current_components = [] (was nil)
- Removed ||= [] lazy init in add_current_component
- Removed &. safe navigation (no longer nullable)
- Removed steep:ignore NoMethod
- Updated RBS: ::Array[Component] (was ::Array[Component]?)

Co-Authored-By: Claude <[email protected]>
@p-datadog
p-datadog merged commit bab5284 into master Apr 22, 2026
592 of 593 checks passed
@p-datadog
p-datadog deleted the di-iseq-backfill branch April 22, 2026 13:48
p-datadog pushed a commit that referenced this pull request Apr 22, 2026
…ent section

The section said "Code loaded before the tracer initializes cannot be
instrumented with line probes." After #5496 (whole-file iseq backfill) and
this PR (per-method iseq fallback), that limitation no longer exists. Remove
the section entirely rather than rewrite it.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos profiling Involves Datadog profiling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants