Skip to content

Add probe status error reporting for snapshot JSON encoding failures#5448

Merged
p-datadog merged 48 commits into
masterfrom
di-snapshot-json-encoding-error-handling
Mar 18, 2026
Merged

Add probe status error reporting for snapshot JSON encoding failures#5448
p-datadog merged 48 commits into
masterfrom
di-snapshot-json-encoding-error-handling

Conversation

@p-datadog

@p-datadog p-datadog commented Mar 13, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Adds proper error handling when snapshot JSON encoding fails due to custom serializers producing non-UTF-8 binary data. When encoding fails, the probe is disabled and ERROR status is reported to the UI.

Motivation:

When custom serializers produce binary strings (ASCII-8BIT with bytes >= 0x80), JSON encoding fails during snapshot transmission. Without this PR, these failures are silent - snapshots are dropped, probes appear healthy, and only telemetry tracks the issue.

Change log entry

Yes. DI: Custom serializers producing non-JSON-encodable data now show ERROR probe status instead of silent failures.

Additional Notes:

Architecture: Uses callback-based design where Transport serializes each snapshot individually and calls on_serialization_error callback on failure. ProbeNotifierWorker provides the callback that disables the probe and reports ERROR status. This keeps serialization concerns in Transport while allowing Worker to handle error responses.

Key changes:

  • ProbeRepository extracted to manage probe storage (installed, pending, failed)
  • ProbeNotifierWorker receives probe_repository and probe_notification_builder at construction
  • Transport::Input#send_input accepts on_serialization_error callback
  • JSON failures are isolated per-snapshot (other snapshots in batch still sent)
  • Added documentation warning about custom serializer requirements

How to test the change?

bundle exec rspec spec/datadog/di/probe_notifier_worker_spec.rb
bundle exec rspec spec/datadog/di/transport/input_spec.rb
bundle exec rspec spec/datadog/di/probe_manager_spec.rb

All DI tests pass.

Verified manually:

1

When custom serializers produce binary data or other non-UTF-8 strings
that cannot be JSON-encoded, the snapshot now fails gracefully with
proper error reporting instead of silent failure.

Changes:
- Catch JSON::GeneratorError in ProbeNotifierWorker.do_send_snapshot
- Extract probe IDs from failed snapshot batch
- Invoke callback to ProbeManager for each failed probe
- Automatically disable probe to prevent repeated failures
- Send ERROR status with specific message about JSON encoding failure
- Log error and report to telemetry

Add documentation warning to custom serializer section explaining that
serializers must produce JSON-encodable data and must not return binary
strings (ASCII-8BIT) with non-ASCII bytes that cannot be transcoded to UTF-8.

Add tests demonstrating binary string JSON encoding failure and verifying
callback invocation with proper error handling.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@p-datadog
p-datadog requested a review from a team as a code owner March 13, 2026 16:55
@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Mar 13, 2026
@pr-commenter

pr-commenter Bot commented Mar 13, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-17 16:44:25

Comparing candidate commit ac2fde5 in PR branch di-snapshot-json-encoding-error-handling with baseline commit 6f300a1 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Resolved conflicts in:
- lib/datadog/di/serializer.rb: Combined documentation about exception handling and JSON encoding requirements
- spec/datadog/di/serializer_spec.rb: Used master's comprehensive binary data escaping tests
@github-actions

github-actions Bot commented Mar 13, 2026

Copy link
Copy Markdown

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 4 steep:ignore comments, and clears 4 steep:ignore comments.

steep:ignore comments (+4-4)Introduced:
lib/datadog/di/boot.rb:22
lib/datadog/di/boot.rb:41
lib/datadog/di/probe_notification_builder.rb:137
lib/datadog/di/serializer.rb:243
Cleared:
lib/datadog/di/boot.rb:21
lib/datadog/di/boot.rb:40
lib/datadog/di/probe_notification_builder.rb:220
lib/datadog/di/serializer.rb:233

Untyped methods

This PR introduces 10 untyped methods and 5 partially typed methods, and clears 15 untyped methods and 3 partially typed methods. It increases the percentage of typed methods from 61.16% to 61.55% (+0.39%).

Untyped methods (+10-15)Introduced:
sig/datadog/di/component.rbs:61
└── def shutdown!: (?untyped? replacement) -> untyped
sig/datadog/di/component.rbs:63
└── def parse_probe_spec_and_notify: (untyped) -> untyped
sig/datadog/di/probe_manager.rbs:28
└── def close: () -> untyped
sig/datadog/di/probe_manager.rbs:30
└── def clear_hooks: () -> untyped
sig/datadog/di/probe_manager.rbs:40
└── def add_probe: (untyped probe) -> untyped
sig/datadog/di/probe_manager.rbs:42
└── def remove_probe: (untyped probe_id) -> untyped
sig/datadog/di/probe_manager.rbs:45
└── def install_pending_method_probes: (untyped cls) -> untyped
sig/datadog/di/probe_manager.rbs:48
└── def install_pending_line_probes: (untyped path) -> untyped
sig/datadog/di/probe_manager.rbs:50
└── def probe_executed_callback: (untyped context) -> untyped
sig/datadog/di/probe_manager.rbs:52
└── def probe_condition_evaluation_failed_callback: (untyped context, untyped expr, untyped exc) -> untyped
Cleared:
sig/datadog/di/component.rbs:57
└── def shutdown!: (?untyped? replacement) -> untyped
sig/datadog/di/component.rbs:59
└── def parse_probe_spec_and_notify: (untyped) -> untyped
sig/datadog/di/probe_manager.rbs:26
└── def initialize: (untyped settings, untyped instrumenter, untyped probe_notification_builder, untyped probe_notifier_worker, untyped logger, ?telemetry: untyped?) -> void
sig/datadog/di/probe_manager.rbs:31
└── def close: () -> untyped
sig/datadog/di/probe_manager.rbs:33
└── def clear_hooks: () -> untyped
sig/datadog/di/probe_manager.rbs:43
└── def installed_probes: () -> untyped
sig/datadog/di/probe_manager.rbs:45
└── def pending_probes: () -> untyped
sig/datadog/di/probe_manager.rbs:46
└── def failed_probes: () -> untyped
sig/datadog/di/probe_manager.rbs:47
└── def add_probe: (untyped probe) -> untyped
sig/datadog/di/probe_manager.rbs:48
└── def remove_probe: (untyped probe_id) -> untyped
sig/datadog/di/probe_manager.rbs:51
└── def install_pending_method_probes: (untyped cls) -> untyped
sig/datadog/di/probe_manager.rbs:54
└── def install_pending_line_probes: (untyped path) -> untyped
sig/datadog/di/probe_manager.rbs:55
└── def probe_executed_callback: (untyped context) -> untyped
sig/datadog/di/probe_manager.rbs:57
└── def probe_condition_evaluation_failed_callback: (untyped context, untyped expr, untyped exc) -> untyped
sig/datadog/di/transport/input.rbs:29
└── def send_input: (untyped payload, untyped tags) -> untyped
Partially typed methods (+5-3)Introduced:
sig/datadog/di/component.rbs:30
└── def self.build: (untyped settings, untyped agent_settings, Core::Logger logger, ?telemetry: untyped?) -> (nil | untyped)
sig/datadog/di/component.rbs:32
└── def self.environment_supported?: (untyped settings, Core::Logger logger) -> (false | true)
sig/datadog/di/component.rbs:34
└── def initialize: (untyped settings, untyped agent_settings, Core::Logger logger, ?code_tracker: untyped?, ?telemetry: untyped?) -> void
sig/datadog/di/probe_manager.rbs:20
└── def initialize: (untyped settings, untyped instrumenter, untyped probe_notification_builder, untyped probe_notifier_worker, untyped logger, ProbeRepository probe_repository, ?telemetry: untyped?) -> void
sig/datadog/di/transport/input.rbs:29
└── def send_input: (untyped payload, untyped tags, on_serialization_error: ^(String, Exception) -> void) -> untyped
Cleared:
sig/datadog/di/component.rbs:28
└── def self.build: (untyped settings, untyped agent_settings, Core::Logger logger, ?telemetry: untyped?) -> (nil | untyped)
sig/datadog/di/component.rbs:30
└── def self.environment_supported?: (untyped settings, Core::Logger logger) -> (false | true)
sig/datadog/di/component.rbs:32
└── def initialize: (untyped settings, untyped agent_settings, Core::Logger logger, ?code_tracker: untyped?, ?telemetry: untyped?) -> void

Untyped other declarations

This PR introduces 14 untyped other declarations, and clears 18 untyped other declarations. It increases the percentage of typed other declarations from 77.34% to 77.64% (+0.3%).

Untyped other declarations (+14-18)Introduced:
sig/datadog/di/component.rbs:36
└── attr_reader settings: untyped
sig/datadog/di/component.rbs:38
└── attr_reader agent_settings: untyped
sig/datadog/di/component.rbs:40
└── attr_reader logger: untyped
sig/datadog/di/component.rbs:42
└── attr_reader telemetry: untyped
sig/datadog/di/component.rbs:48
└── attr_reader transport: untyped
sig/datadog/di/component.rbs:60
└── attr_reader serializer: untyped
sig/datadog/di/probe_manager.rbs:18
└── @definition_trace_point: untyped
sig/datadog/di/probe_manager.rbs:22
└── attr_reader logger: untyped
sig/datadog/di/probe_manager.rbs:24
└── attr_reader telemetry: untyped
sig/datadog/di/probe_manager.rbs:32
└── attr_reader settings: untyped
sig/datadog/di/probe_manager.rbs:34
└── attr_reader instrumenter: untyped
sig/datadog/di/probe_manager.rbs:36
└── attr_reader probe_notification_builder: untyped
sig/datadog/di/probe_manager.rbs:38
└── attr_reader probe_notifier_worker: untyped
sig/datadog/di/probe_manager.rbs:56
└── attr_reader definition_trace_point: untyped
Cleared:
sig/datadog/di/component.rbs:34
└── attr_reader settings: untyped
sig/datadog/di/component.rbs:36
└── attr_reader agent_settings: untyped
sig/datadog/di/component.rbs:38
└── attr_reader logger: untyped
sig/datadog/di/component.rbs:40
└── attr_reader telemetry: untyped
sig/datadog/di/component.rbs:46
└── attr_reader transport: untyped
sig/datadog/di/component.rbs:56
└── attr_reader serializer: untyped
sig/datadog/di/probe_manager.rbs:16
└── @installed_probes: untyped
sig/datadog/di/probe_manager.rbs:18
└── @pending_probes: untyped
sig/datadog/di/probe_manager.rbs:20
└── @failed_probes: untyped
sig/datadog/di/probe_manager.rbs:22
└── @lock: untyped
sig/datadog/di/probe_manager.rbs:24
└── @definition_trace_point: untyped
sig/datadog/di/probe_manager.rbs:28
└── attr_reader logger: untyped
sig/datadog/di/probe_manager.rbs:30
└── attr_reader telemetry: untyped
sig/datadog/di/probe_manager.rbs:35
└── attr_reader settings: untyped
sig/datadog/di/probe_manager.rbs:37
└── attr_reader instrumenter: untyped
sig/datadog/di/probe_manager.rbs:39
└── attr_reader probe_notification_builder: untyped
sig/datadog/di/probe_manager.rbs:41
└── attr_reader probe_notifier_worker: untyped
sig/datadog/di/probe_manager.rbs:61
└── attr_reader definition_trace_point: untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Mar 13, 2026

Copy link
Copy Markdown

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 95.63%
Overall Coverage: 95.16% (-0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ac2fde5 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

Add RBS type signatures for snapshot_serialization_failed_callback
method and attribute accessor. This resolves type checking errors in
probe_manager.rb and component.rb.

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Comment thread lib/datadog/di/probe_notifier_worker.rb Outdated
Unicorn Enterprises and others added 5 commits March 13, 2026 16:17
Extract ProbeRepository class to break circular dependency between
ProbeNotifierWorker and ProbeManager. This allows all dependencies
to be injected at construction time, eliminating post-creation mutation.

Changes:
- Add ProbeRepository class to manage probe storage (installed, pending, failed)
- Update ProbeManager to delegate probe storage to ProbeRepository
- Update ProbeNotifierWorker to handle JSON serialization errors directly
  using injected probe_repository and probe_notification_builder
- Remove snapshot_serialization_failed_callback attribute and attr_writer
- Update Component to create ProbeRepository first and share it between
  ProbeNotifierWorker and ProbeManager
- Update RBS type definitions for all changed classes
- Update tests to use dependency injection

This is a cleaner dependency injection approach where all dependencies
are injected at construction time - no post-creation wiring needed.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
- ProbeRepository.add_failed now takes (probe_id, message) to store
  error message strings, matching original ProbeManager behavior
- ProbeRepository methods return the actual hash (not .dup()) for
  backward compatibility with existing test assertions
- ProbeRepository RBS updated to reflect String values in failed_probes
- Remove argument from probe.disable!() call since the method doesn't
  accept parameters
- Update probe_notifier_worker tests to properly mock instance_double
  methods and use .send() interception

Co-Authored-By: Claude Opus 4.5 <[email protected]>
The failed_probes hash stores error message strings (not Probe objects),
so the RBS return type should be Hash[String, String].

Co-Authored-By: Claude Opus 4.5 <[email protected]>
The code is self-explanatory for disabling the probe and reporting status.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
When a probe fails to install (e.g., due to instrumentation error or
already-installed error), report ERROR status to the agent so it shows
up in the UI. Previously, failures were only logged and tracked in
telemetry.

This addresses the TODO comment that was in the code.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Comment thread lib/datadog/di/probe_notifier_worker.rb Outdated
Comment thread lib/datadog/di/probe_notifier_worker.rb Outdated
Comment thread lib/datadog/di/probe_repository.rb Outdated
Comment thread lib/datadog/di/probe_repository.rb Outdated
Comment thread lib/datadog/di/probe_repository.rb Outdated
Comment thread lib/datadog/di/probe_repository.rb Outdated
Comment thread lib/datadog/di/probe_repository.rb Outdated
Comment thread lib/datadog/di/probe_repository.rb Outdated
Unicorn Enterprises and others added 3 commits March 13, 2026 17:58
When a pending probe fails to install after its class is defined,
report ERROR status to the agent and add to failed_probes list.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review comments:
- Remove "avoiding circular dependencies" from comments
- Document batch parameter in handle_snapshot_serialization_failure
- Clarify find_failed looks up by probe ID
- Clarify failed_probes stores error messages, not probe objects
- Explain when remove_failed is called
- Add yield/lock safety analysis for clear_all method
- Improve class-level documentation describing the three collections

- Fixed in lib/datadog/di/probe_repository.rb (class docs, find_failed, add_failed, remove_failed, clear_all)
- Fixed in lib/datadog/di/probe_notifier_worker.rb (handle_snapshot_serialization_failure docs)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review comments:
- Clarify that yield only happens when block is given
- Move yield outside the lock to avoid blocking other operations

The probes are copied and cleared atomically under the lock, then
the cleanup callback is invoked outside the lock. This prevents
blocking if unhook is slow.

- Fixed in lib/datadog/di/probe_repository.rb:clear_all

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Comment thread lib/datadog/di/component.rb Outdated
Comment thread lib/datadog/di/component.rb Outdated
Comment thread lib/datadog/di/component.rb Outdated
Comment thread lib/datadog/di/component.rb Outdated
Address review comments: Remove comments that just restate
what the code does ("Create shared probe repository first",
"Create notification builder", etc.)

- Fixed in lib/datadog/di/component.rb:66-78

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Comment thread lib/datadog/di/probe_manager.rb Outdated
Unicorn Enterprises and others added 4 commits March 13, 2026 18:17
Add comprehensive tests for the new ProbeRepository class:
- installed_probes: find, add, remove operations
- pending_probes: find, add, remove, clear operations
- failed_probes: find, add, remove operations
- clear_all: with and without block, verifies yield behavior

23 examples, 0 failures

Co-Authored-By: Claude Opus 4.5 <[email protected]>
The flush method blocks until queues are empty and IO is complete.
Error handling happens synchronously in do_send_snapshot, so flush
is sufficient - no additional sleep needed.

Verified tests pass 5x in a row without sleep (deterministic).

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Since DI is @api private, no backward compatibility is needed.
Remove installed_probes, pending_probes, failed_probes delegation
methods from ProbeManager - callers should access probe_repository
directly.

- Removed 3 delegation methods from lib/datadog/di/probe_manager.rb
- Updated RBS file
- Updated tests to use manager.probe_repository.* instead

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Fix integration tests to use probe_repository directly after removing
installed_probes, pending_probes, and failed_probes delegation methods
from ProbeManager.

- Updated spec/datadog/di/integration/instrumentation_spec.rb
- Updated spec/datadog/di/integration/everything_from_remote_config_spec.rb

Changed all occurrences:
- probe_manager.installed_probes → probe_manager.probe_repository.installed_probes
- probe_manager.pending_probes → probe_manager.probe_repository.pending_probes

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Comment thread lib/datadog/di/probe_manager.rb Outdated
Address review comment: Change probe_repository from optional keyword
parameter to required positional parameter.

Changes:
- lib/datadog/di/probe_manager.rb:19-20 - Made probe_repository required positional parameter
- lib/datadog/di/probe_manager.rb:27 - Removed fallback to ProbeRepository.new
- lib/datadog/di/component.rb:75-77 - Pass probe_repository as positional parameter
- sig/datadog/di/probe_manager.rbs:20 - Updated type signature
- spec/datadog/di/probe_manager_spec.rb:30-34 - Added probe_repository let block and pass to manager
- spec/datadog/di/remote_spec.rb:115-120 - Added probe_repository let block and pass to probe_manager
- spec/datadog/di/instrumenter_circuit_breaker_spec.rb:373-380 - Added probe_repository let block and pass to probe_manager

All tests pass:
- 18 examples in probe_manager_spec.rb
- 6 examples in remote_spec.rb
- 7 examples in instrumenter_circuit_breaker_spec.rb
- 4 examples in component_spec.rb
- Type checking passes (steep check)

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Comment thread spec/datadog/di/probe_manager_spec.rb Outdated
Address review comment: Replace all manager.probe_repository references
with the probe_repository let variable for consistency and clarity.

Changes:
- spec/datadog/di/probe_manager_spec.rb - Replaced 31 occurrences of
  manager.probe_repository with probe_repository

This improves test readability by using the let variable directly
instead of accessing it through the manager.

All tests pass:
- 18 examples in probe_manager_spec.rb

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Comment thread spec/datadog/di/probe_notifier_worker_spec.rb Outdated
Unicorn Enterprises and others added 3 commits March 15, 2026 11:39
Reference Transport::Input::Transport constants for snapshot size limits
and chunk sizes instead of duplicating them in ProbeNotifierWorker.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
…dling

Previously, when a batch of snapshots contained one that failed JSON
encoding, all probes in the batch were disabled. This was incorrect
because healthy probes were being disabled due to failures in unrelated
probes.

This refactors the design to:
1. Transport serializes each snapshot individually, isolating failures
2. Transport calls on_serialization_error callback when serialization fails
3. ProbeNotifierWorker provides callback that disables the affected probe
   and sends ERROR status to the backend
4. ProbeManager sends Hash objects (not JSON strings), maintaining clean
   separation of concerns

The callback-based design keeps serialization in the transport layer where
it belongs, while allowing the caller to handle failures appropriately.

- Transport.send_input now accepts on_serialization_error callback
- ProbeNotifierWorker.handle_serialization_error disables probe and sends
  ERROR status when serialization fails
- Component passes probe_repository and probe_notification_builder to
  worker for error handling
- ProbeManager simplified to just pass Hash payloads

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add the on_serialization_error keyword parameter to the send_input
method signature in the RBS type definitions.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Comment thread lib/datadog/di/probe_manager.rb Outdated
Unicorn Enterprises and others added 11 commits March 15, 2026 12:49
- Clarify rate limiting explanation for probe_condition_evaluation_failed_callback
  (token bucket algorithm with burst and refill)
- Fix probe_disabled_callback to describe CPU time-based circuit breaker
  rather than rate limiting

Co-Authored-By: Claude Opus 4.5 <[email protected]>
When binary data (ASCII-8BIT) is included in snapshot data, JSON.dump
raises Encoding::UndefinedConversionError rather than JSON::GeneratorError.
This can happen when custom serializers return binary strings.

Add Encoding::UndefinedConversionError to the rescue clause so that
encoding failures from binary data are properly caught and reported
via the on_serialization_error callback.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Binary data in JSON encoding can raise either JSON::GeneratorError or
Encoding::UndefinedConversionError depending on Ruby version and how
JSON.dump handles the encoding. Update test expectations to accept
either exception type.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Use a blanket rescue clause to catch all possible serialization errors,
not just JSON::GeneratorError and Encoding::UndefinedConversionError.
This handles edge cases like:
- Encoding::InvalidByteSequenceError
- TypeError (non-serializable objects like Proc, IO)
- SystemStackError (infinite recursion)
- Errors from custom to_json implementations

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review feedback for PR #5448:

- Make probe_repository and probe_notification_builder required keyword
  args in ProbeNotifierWorker (production always provides them)
- Remove nil guard clause in handle_serialization_error
- Add @return/@param/@raise YARD tags to ProbeRepository, ProbeManager,
  and ProbeNotifierWorker public methods
- Replace bare allow(logger).to receive(:debug) with expect_lazy_log
  that verifies message content (error details, probe IDs)
- Add dedicated test for serialization error log message
- Remove test for nil probe_repository (no longer applicable)
- Update RBS signatures to match non-optional types
- Reorder keyword params per StandardRB (required before optional)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Wrap the on_serialization_error callback invocation in a rescue block
so that exceptions from the callback do not break iteration over
remaining snapshots in the batch. This ensures that a failure in the
error-handling path (e.g., probe disabling or status reporting) does
not prevent other healthy snapshots from being sent.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Move build_status above the private keyword so it can be called
directly instead of via send(:build_status, ...). This method is
the general-purpose status builder used by all other public build_*
methods, and is now also called directly by ProbeNotifierWorker for
serialization error reporting.

- Add YARD documentation to build_status
- Update ProbeNotifierWorker to call build_status directly
- Update test expectations from receive(:send) to receive(:build_status)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add synchronize method to ProbeRepository exposing the Monitor lock
for compound operations. Wrap add_probe, install_pending_method_probes,
and install_pending_line_probes in probe_repository.synchronize to
ensure atomicity of check-then-act sequences.

This restores the thread-safety properties of the old code where the
entire add_probe flow (check installed → check failed → hook → store)
was under a single lock. The Monitor is reentrant, so individual
ProbeRepository methods called within the synchronized block safely
re-acquire the lock.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The comment claimed the rate limiter "allows an initial burst of
notifications, then refills tokens over time." In reality, the
TokenBucket is initialized with rate=1, max_tokens=1, which is a
simple 1-per-second limit with no burst capacity. Simplified to
accurately describe the behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Update probe_notifier_worker integration spec to provide the
now-required probe_repository and probe_notification_builder
keyword arguments.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Comment thread lib/datadog/di/probe_notifier_worker.rb
The only production caller (ProbeNotifierWorker#do_send_snapshot)
always provides the callback. The nil default existed only for test
convenience. Make it required and update tests to provide a no-op
handler.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Comment thread lib/datadog/di/probe_notifier_worker.rb Outdated
Comment thread lib/datadog/di/probe_notifier_worker.rb Outdated
Comment thread lib/datadog/di/transport/input.rb
- Simplify probe_disabled_callback docstring to just mention CPU
  consumption limit without detailed circuit breaker description
- Use exception instead of exception.message in log and status
  messages (per reviewer suggestions)
- Add comment explaining why find_installed is used in
  handle_serialization_error (only installed probes produce snapshots)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Comment thread spec/datadog/di/transport/input_spec.rb
Unicorn Enterprises and others added 2 commits March 15, 2026 22:03
Two coverage gaps identified in review:

1. ProbeManager#add_probe had an explicitly TODO-marked untested path:
   when a probe previously failed to install and is re-added,
   ProbePreviouslyFailed is raised through the error handling rescue
   which reports ERROR status. Add a test covering this full path.

2. The core feature (snapshot serialization fails → probe disabled →
   ERROR status sent) was only tested via two separate unit tests.
   Add an end-to-end test that runs the real transport serialization
   logic (with send_input_chunk stubbed to avoid network calls) and
   verifies the full chain: binary snapshot → callback fired → probe
   disabled + ERROR status queued.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Resolve conflict in spec/datadog/di/serializer_spec.rb: keep both
DISerializerBinaryTestClass (from this branch) and
DISerializerStackOverflowTestClass/DISerializerOutOfMemoryTestClass
(from master).

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Comment thread spec/datadog/di/probe_notifier_worker_spec.rb Outdated
Address review comment: the test was stubbing build_status and then only
verifying build_status was called, never checking the return value reached
the transport. Replace with an expectation on send_diagnostics to verify
the status payload flows end-to-end.

- Fixed in spec/datadog/di/probe_notifier_worker_spec.rb:391

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@p-datadog
p-datadog merged commit 78f9609 into master Mar 18, 2026
630 of 631 checks passed
@p-datadog
p-datadog deleted the di-snapshot-json-encoding-error-handling branch March 18, 2026 16:02
@github-actions github-actions Bot added this to the 2.30.0 milestone Mar 18, 2026
@Strech Strech mentioned this pull request Mar 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants