Add probe status error reporting for snapshot JSON encoding failures#5448
Conversation
When custom serializers produce binary data or other non-UTF-8 strings that cannot be JSON-encoded, the snapshot now fails gracefully with proper error reporting instead of silent failure. Changes: - Catch JSON::GeneratorError in ProbeNotifierWorker.do_send_snapshot - Extract probe IDs from failed snapshot batch - Invoke callback to ProbeManager for each failed probe - Automatically disable probe to prevent repeated failures - Send ERROR status with specific message about JSON encoding failure - Log error and report to telemetry Add documentation warning to custom serializer section explaining that serializers must produce JSON-encodable data and must not return binary strings (ASCII-8BIT) with non-ASCII bytes that cannot be transcoded to UTF-8. Add tests demonstrating binary string JSON encoding failure and verifying callback invocation with proper error handling. Co-Authored-By: Claude Opus 4.5 <[email protected]>
BenchmarksBenchmark execution time: 2026-03-17 16:44:25 Comparing candidate commit ac2fde5 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.
|
Resolved conflicts in: - lib/datadog/di/serializer.rb: Combined documentation about exception handling and JSON encoding requirements - spec/datadog/di/serializer_spec.rb: Used master's comprehensive binary data escaping tests
Typing analysisNote: Ignored files are excluded from the next sections.
|
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: ac2fde5 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
Add RBS type signatures for snapshot_serialization_failed_callback method and attribute accessor. This resolves type checking errors in probe_manager.rb and component.rb. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Extract ProbeRepository class to break circular dependency between ProbeNotifierWorker and ProbeManager. This allows all dependencies to be injected at construction time, eliminating post-creation mutation. Changes: - Add ProbeRepository class to manage probe storage (installed, pending, failed) - Update ProbeManager to delegate probe storage to ProbeRepository - Update ProbeNotifierWorker to handle JSON serialization errors directly using injected probe_repository and probe_notification_builder - Remove snapshot_serialization_failed_callback attribute and attr_writer - Update Component to create ProbeRepository first and share it between ProbeNotifierWorker and ProbeManager - Update RBS type definitions for all changed classes - Update tests to use dependency injection This is a cleaner dependency injection approach where all dependencies are injected at construction time - no post-creation wiring needed. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- ProbeRepository.add_failed now takes (probe_id, message) to store error message strings, matching original ProbeManager behavior - ProbeRepository methods return the actual hash (not .dup()) for backward compatibility with existing test assertions - ProbeRepository RBS updated to reflect String values in failed_probes - Remove argument from probe.disable!() call since the method doesn't accept parameters - Update probe_notifier_worker tests to properly mock instance_double methods and use .send() interception Co-Authored-By: Claude Opus 4.5 <[email protected]>
The failed_probes hash stores error message strings (not Probe objects), so the RBS return type should be Hash[String, String]. Co-Authored-By: Claude Opus 4.5 <[email protected]>
The code is self-explanatory for disabling the probe and reporting status. Co-Authored-By: Claude Opus 4.5 <[email protected]>
When a probe fails to install (e.g., due to instrumentation error or already-installed error), report ERROR status to the agent so it shows up in the UI. Previously, failures were only logged and tracked in telemetry. This addresses the TODO comment that was in the code. Co-Authored-By: Claude Opus 4.5 <[email protected]>
When a pending probe fails to install after its class is defined, report ERROR status to the agent and add to failed_probes list. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review comments: - Remove "avoiding circular dependencies" from comments - Document batch parameter in handle_snapshot_serialization_failure - Clarify find_failed looks up by probe ID - Clarify failed_probes stores error messages, not probe objects - Explain when remove_failed is called - Add yield/lock safety analysis for clear_all method - Improve class-level documentation describing the three collections - Fixed in lib/datadog/di/probe_repository.rb (class docs, find_failed, add_failed, remove_failed, clear_all) - Fixed in lib/datadog/di/probe_notifier_worker.rb (handle_snapshot_serialization_failure docs) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review comments: - Clarify that yield only happens when block is given - Move yield outside the lock to avoid blocking other operations The probes are copied and cleared atomically under the lock, then the cleanup callback is invoked outside the lock. This prevents blocking if unhook is slow. - Fixed in lib/datadog/di/probe_repository.rb:clear_all Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review comments: Remove comments that just restate
what the code does ("Create shared probe repository first",
"Create notification builder", etc.)
- Fixed in lib/datadog/di/component.rb:66-78
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add comprehensive tests for the new ProbeRepository class: - installed_probes: find, add, remove operations - pending_probes: find, add, remove, clear operations - failed_probes: find, add, remove operations - clear_all: with and without block, verifies yield behavior 23 examples, 0 failures Co-Authored-By: Claude Opus 4.5 <[email protected]>
The flush method blocks until queues are empty and IO is complete. Error handling happens synchronously in do_send_snapshot, so flush is sufficient - no additional sleep needed. Verified tests pass 5x in a row without sleep (deterministic). Co-Authored-By: Claude Opus 4.5 <[email protected]>
Since DI is @api private, no backward compatibility is needed. Remove installed_probes, pending_probes, failed_probes delegation methods from ProbeManager - callers should access probe_repository directly. - Removed 3 delegation methods from lib/datadog/di/probe_manager.rb - Updated RBS file - Updated tests to use manager.probe_repository.* instead Co-Authored-By: Claude Opus 4.5 <[email protected]>
Fix integration tests to use probe_repository directly after removing installed_probes, pending_probes, and failed_probes delegation methods from ProbeManager. - Updated spec/datadog/di/integration/instrumentation_spec.rb - Updated spec/datadog/di/integration/everything_from_remote_config_spec.rb Changed all occurrences: - probe_manager.installed_probes → probe_manager.probe_repository.installed_probes - probe_manager.pending_probes → probe_manager.probe_repository.pending_probes Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Address review comment: Change probe_repository from optional keyword parameter to required positional parameter. Changes: - lib/datadog/di/probe_manager.rb:19-20 - Made probe_repository required positional parameter - lib/datadog/di/probe_manager.rb:27 - Removed fallback to ProbeRepository.new - lib/datadog/di/component.rb:75-77 - Pass probe_repository as positional parameter - sig/datadog/di/probe_manager.rbs:20 - Updated type signature - spec/datadog/di/probe_manager_spec.rb:30-34 - Added probe_repository let block and pass to manager - spec/datadog/di/remote_spec.rb:115-120 - Added probe_repository let block and pass to probe_manager - spec/datadog/di/instrumenter_circuit_breaker_spec.rb:373-380 - Added probe_repository let block and pass to probe_manager All tests pass: - 18 examples in probe_manager_spec.rb - 6 examples in remote_spec.rb - 7 examples in instrumenter_circuit_breaker_spec.rb - 4 examples in component_spec.rb - Type checking passes (steep check) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Address review comment: Replace all manager.probe_repository references with the probe_repository let variable for consistency and clarity. Changes: - spec/datadog/di/probe_manager_spec.rb - Replaced 31 occurrences of manager.probe_repository with probe_repository This improves test readability by using the let variable directly instead of accessing it through the manager. All tests pass: - 18 examples in probe_manager_spec.rb Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Reference Transport::Input::Transport constants for snapshot size limits and chunk sizes instead of duplicating them in ProbeNotifierWorker. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…dling Previously, when a batch of snapshots contained one that failed JSON encoding, all probes in the batch were disabled. This was incorrect because healthy probes were being disabled due to failures in unrelated probes. This refactors the design to: 1. Transport serializes each snapshot individually, isolating failures 2. Transport calls on_serialization_error callback when serialization fails 3. ProbeNotifierWorker provides callback that disables the affected probe and sends ERROR status to the backend 4. ProbeManager sends Hash objects (not JSON strings), maintaining clean separation of concerns The callback-based design keeps serialization in the transport layer where it belongs, while allowing the caller to handle failures appropriately. - Transport.send_input now accepts on_serialization_error callback - ProbeNotifierWorker.handle_serialization_error disables probe and sends ERROR status when serialization fails - Component passes probe_repository and probe_notification_builder to worker for error handling - ProbeManager simplified to just pass Hash payloads Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add the on_serialization_error keyword parameter to the send_input method signature in the RBS type definitions. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Clarify rate limiting explanation for probe_condition_evaluation_failed_callback (token bucket algorithm with burst and refill) - Fix probe_disabled_callback to describe CPU time-based circuit breaker rather than rate limiting Co-Authored-By: Claude Opus 4.5 <[email protected]>
When binary data (ASCII-8BIT) is included in snapshot data, JSON.dump raises Encoding::UndefinedConversionError rather than JSON::GeneratorError. This can happen when custom serializers return binary strings. Add Encoding::UndefinedConversionError to the rescue clause so that encoding failures from binary data are properly caught and reported via the on_serialization_error callback. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Binary data in JSON encoding can raise either JSON::GeneratorError or Encoding::UndefinedConversionError depending on Ruby version and how JSON.dump handles the encoding. Update test expectations to accept either exception type. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Use a blanket rescue clause to catch all possible serialization errors, not just JSON::GeneratorError and Encoding::UndefinedConversionError. This handles edge cases like: - Encoding::InvalidByteSequenceError - TypeError (non-serializable objects like Proc, IO) - SystemStackError (infinite recursion) - Errors from custom to_json implementations Co-Authored-By: Claude Opus 4.5 <[email protected]>
Address review feedback for PR #5448: - Make probe_repository and probe_notification_builder required keyword args in ProbeNotifierWorker (production always provides them) - Remove nil guard clause in handle_serialization_error - Add @return/@param/@raise YARD tags to ProbeRepository, ProbeManager, and ProbeNotifierWorker public methods - Replace bare allow(logger).to receive(:debug) with expect_lazy_log that verifies message content (error details, probe IDs) - Add dedicated test for serialization error log message - Remove test for nil probe_repository (no longer applicable) - Update RBS signatures to match non-optional types - Reorder keyword params per StandardRB (required before optional) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Wrap the on_serialization_error callback invocation in a rescue block so that exceptions from the callback do not break iteration over remaining snapshots in the batch. This ensures that a failure in the error-handling path (e.g., probe disabling or status reporting) does not prevent other healthy snapshots from being sent. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Move build_status above the private keyword so it can be called directly instead of via send(:build_status, ...). This method is the general-purpose status builder used by all other public build_* methods, and is now also called directly by ProbeNotifierWorker for serialization error reporting. - Add YARD documentation to build_status - Update ProbeNotifierWorker to call build_status directly - Update test expectations from receive(:send) to receive(:build_status) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add synchronize method to ProbeRepository exposing the Monitor lock for compound operations. Wrap add_probe, install_pending_method_probes, and install_pending_line_probes in probe_repository.synchronize to ensure atomicity of check-then-act sequences. This restores the thread-safety properties of the old code where the entire add_probe flow (check installed → check failed → hook → store) was under a single lock. The Monitor is reentrant, so individual ProbeRepository methods called within the synchronized block safely re-acquire the lock. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The comment claimed the rate limiter "allows an initial burst of notifications, then refills tokens over time." In reality, the TokenBucket is initialized with rate=1, max_tokens=1, which is a simple 1-per-second limit with no burst capacity. Simplified to accurately describe the behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Update probe_notifier_worker integration spec to provide the now-required probe_repository and probe_notification_builder keyword arguments. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The only production caller (ProbeNotifierWorker#do_send_snapshot) always provides the callback. The nil default existed only for test convenience. Make it required and update tests to provide a no-op handler. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Simplify probe_disabled_callback docstring to just mention CPU consumption limit without detailed circuit breaker description - Use exception instead of exception.message in log and status messages (per reviewer suggestions) - Add comment explaining why find_installed is used in handle_serialization_error (only installed probes produce snapshots) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Two coverage gaps identified in review: 1. ProbeManager#add_probe had an explicitly TODO-marked untested path: when a probe previously failed to install and is re-added, ProbePreviouslyFailed is raised through the error handling rescue which reports ERROR status. Add a test covering this full path. 2. The core feature (snapshot serialization fails → probe disabled → ERROR status sent) was only tested via two separate unit tests. Add an end-to-end test that runs the real transport serialization logic (with send_input_chunk stubbed to avoid network calls) and verifies the full chain: binary snapshot → callback fired → probe disabled + ERROR status queued. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Resolve conflict in spec/datadog/di/serializer_spec.rb: keep both DISerializerBinaryTestClass (from this branch) and DISerializerStackOverflowTestClass/DISerializerOutOfMemoryTestClass (from master). Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Address review comment: the test was stubbing build_status and then only verifying build_status was called, never checking the return value reached the transport. Replace with an expectation on send_diagnostics to verify the status payload flows end-to-end. - Fixed in spec/datadog/di/probe_notifier_worker_spec.rb:391 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
What does this PR do?
Adds proper error handling when snapshot JSON encoding fails due to custom serializers producing non-UTF-8 binary data. When encoding fails, the probe is disabled and ERROR status is reported to the UI.
Motivation:
When custom serializers produce binary strings (ASCII-8BIT with bytes >= 0x80), JSON encoding fails during snapshot transmission. Without this PR, these failures are silent - snapshots are dropped, probes appear healthy, and only telemetry tracks the issue.
Change log entry
Yes. DI: Custom serializers producing non-JSON-encodable data now show ERROR probe status instead of silent failures.
Additional Notes:
Architecture: Uses callback-based design where Transport serializes each snapshot individually and calls
on_serialization_errorcallback on failure. ProbeNotifierWorker provides the callback that disables the probe and reports ERROR status. This keeps serialization concerns in Transport while allowing Worker to handle error responses.Key changes:
ProbeRepositoryextracted to manage probe storage (installed, pending, failed)ProbeNotifierWorkerreceivesprobe_repositoryandprobe_notification_builderat constructionTransport::Input#send_inputacceptson_serialization_errorcallbackHow to test the change?
All DI tests pass.
Verified manually: