fix(guardrails): forward abort telemetry synchronously to avoid a hung exit#9268
Conversation
…g exit Single-step install injects the tracer via NODE_OPTIONS. When it bails out on an incompatible runtime it fires abort telemetry by spawning the forwarder with child_process.spawn, then the injected app returns and typically calls process.exit(). On Node 24.0.0/24.1.x a child still tearing down its stdio pipes deadlocks process.exit() (fixed upstream in 24.2), so the forwarder keeps the short-lived process alive until it is killed externally. Because injection is automatic, an unsupported-runtime customer with short-lived processes gets hangs. Forward the abort telemetry with spawnSync so the child is fully reaped before we return and nothing outlives the call to race the exit. The synchronous path runs only on the non-forced bailout, before any instrumentation is active, so the forwarder is never traced; the async spawn stays for the post-init `complete` point. spawnSync exists since Node 0.11.12 and the guardrails still target >=0.8, which predates the exit bug, so the async path remains the fallback there. Refs: nodejs/node#63085
Overall package sizeSelf size: 6.69 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 437.94 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
| initBailout = true | ||
| var runtimeInfo = 'Incompatible runtime Node.js ' + version + ', supported runtimes: Node.js ' + supportedRange | ||
| // When not forced, the process bails out here and may call process.exit() right away; | ||
| // forward synchronously so the telemetry child can't outlive us and wedge the exit. |
There was a problem hiding this comment.
Shouldn't this be handled properly at the source instead of worked around here?
There was a problem hiding this comment.
The source being Node.js? This issue is fixed in newer Node.js versions
There was a problem hiding this comment.
Ah, the comment wasn't clear that the underlying issue is outside the library, but I see it in another comment 👍
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6717dfb843
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| var result = spawnSync(telemetryForwarderPath, ['library_entrypoint'], { | ||
| input: payload, | ||
| stdio: ['pipe', 'ignore', 'ignore'] |
There was a problem hiding this comment.
Add a timeout to the synchronous forwarder
On the unsupported-runtime SSI path, this waits forever for whatever executable DD_TELEMETRY_FORWARDER_PATH names; the only validation above is existsSync, so an existing forwarder or wrapper that wedges or blocks before exiting now hangs the user's process before guardrails return and before the app can reach its own process.exit(). Since this telemetry is best-effort and this change is specifically avoiding hangs, please bound the synchronous send with a short timeout/kill handling or fall back so a bad forwarder cannot become a new hard hang.
Useful? React with 👍 / 👎.
BenchmarksBenchmark execution time: 2026-07-09 17:47:05 Comparing candidate commit 171a96b in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2321 metrics, 37 unstable metrics.
|
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 171a96b | Docs | Datadog PR Page | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9268 +/- ##
========================================
Coverage 96.56% 96.56%
========================================
Files 918 918
Lines 121521 121560 +39
Branches 21522 21004 -518
========================================
+ Hits 117347 117390 +43
+ Misses 4174 4170 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…e exit The synchronous bailout path validates only that DD_TELEMETRY_FORWARDER_PATH exists, then blocks on spawnSync with no timeout. A forwarder or wrapper that wedges before exiting would hang the injected process before guardrails return and before the app can reach its own process.exit() — reintroducing the exact hard hang this change exists to remove. Bound the send with a 1s timeout and SIGKILL, and log ETIMEDOUT distinctly; the telemetry is best-effort, so a bad forwarder degrades to a dropped point instead of a stuck process.
The spawnSync bailout path logs two failures that had no coverage: a non-ETIMEDOUT spawn error (a forwarder that exists but cannot execute, e.g. EACCES) and a non-zero forwarder exit code. Both are reachable and mirror the async path's handling, so they get real cases beside the existing ETIMEDOUT test using the same mocked-spawnSync pattern.
…g exit (#9268) * fix(guardrails): forward abort telemetry synchronously to avoid a hung exit Single-step install injects the tracer via NODE_OPTIONS. When it bails out on an incompatible runtime it fires abort telemetry by spawning the forwarder with child_process.spawn, then the injected app returns and typically calls process.exit(). On Node 24.0.0/24.1.x a child still tearing down its stdio pipes deadlocks process.exit() (fixed upstream in 24.2), so the forwarder keeps the short-lived process alive until it is killed externally. Because injection is automatic, an unsupported-runtime customer with short-lived processes gets hangs. Forward the abort telemetry with spawnSync so the child is fully reaped before we return and nothing outlives the call to race the exit. The synchronous path runs only on the non-forced bailout, before any instrumentation is active, so the forwarder is never traced; the async spawn stays for the post-init `complete` point. spawnSync exists since Node 0.11.12 and the guardrails still target >=0.8, which predates the exit bug, so the async path remains the fallback there. Refs: nodejs/node#63085 * fix(guardrails): bound the synchronous forwarder so it cannot hang the exit The synchronous bailout path validates only that DD_TELEMETRY_FORWARDER_PATH exists, then blocks on spawnSync with no timeout. A forwarder or wrapper that wedges before exiting would hang the injected process before guardrails return and before the app can reach its own process.exit() — reintroducing the exact hard hang this change exists to remove. Bound the send with a 1s timeout and SIGKILL, and log ETIMEDOUT distinctly; the telemetry is best-effort, so a bad forwarder degrades to a dropped point instead of a stuck process. * test(guardrails): cover the synchronous forwarder error branches The spawnSync bailout path logs two failures that had no coverage: a non-ETIMEDOUT spawn error (a forwarder that exists but cannot execute, e.g. EACCES) and a non-zero forwarder exit code. Both are reachable and mirror the async path's handling, so they get real cases beside the existing ETIMEDOUT test using the same mocked-spawnSync pattern.
…g exit (#9268) * fix(guardrails): forward abort telemetry synchronously to avoid a hung exit Single-step install injects the tracer via NODE_OPTIONS. When it bails out on an incompatible runtime it fires abort telemetry by spawning the forwarder with child_process.spawn, then the injected app returns and typically calls process.exit(). On Node 24.0.0/24.1.x a child still tearing down its stdio pipes deadlocks process.exit() (fixed upstream in 24.2), so the forwarder keeps the short-lived process alive until it is killed externally. Because injection is automatic, an unsupported-runtime customer with short-lived processes gets hangs. Forward the abort telemetry with spawnSync so the child is fully reaped before we return and nothing outlives the call to race the exit. The synchronous path runs only on the non-forced bailout, before any instrumentation is active, so the forwarder is never traced; the async spawn stays for the post-init `complete` point. spawnSync exists since Node 0.11.12 and the guardrails still target >=0.8, which predates the exit bug, so the async path remains the fallback there. Refs: nodejs/node#63085 * fix(guardrails): bound the synchronous forwarder so it cannot hang the exit The synchronous bailout path validates only that DD_TELEMETRY_FORWARDER_PATH exists, then blocks on spawnSync with no timeout. A forwarder or wrapper that wedges before exiting would hang the injected process before guardrails return and before the app can reach its own process.exit() — reintroducing the exact hard hang this change exists to remove. Bound the send with a 1s timeout and SIGKILL, and log ETIMEDOUT distinctly; the telemetry is best-effort, so a bad forwarder degrades to a dropped point instead of a stuck process. * test(guardrails): cover the synchronous forwarder error branches The spawnSync bailout path logs two failures that had no coverage: a non-ETIMEDOUT spawn error (a forwarder that exists but cannot execute, e.g. EACCES) and a non-zero forwarder exit code. Both are reachable and mirror the async path's handling, so they get real cases beside the existing ETIMEDOUT test using the same mocked-spawnSync pattern.
…g exit (#9268) * fix(guardrails): forward abort telemetry synchronously to avoid a hung exit Single-step install injects the tracer via NODE_OPTIONS. When it bails out on an incompatible runtime it fires abort telemetry by spawning the forwarder with child_process.spawn, then the injected app returns and typically calls process.exit(). On Node 24.0.0/24.1.x a child still tearing down its stdio pipes deadlocks process.exit() (fixed upstream in 24.2), so the forwarder keeps the short-lived process alive until it is killed externally. Because injection is automatic, an unsupported-runtime customer with short-lived processes gets hangs. Forward the abort telemetry with spawnSync so the child is fully reaped before we return and nothing outlives the call to race the exit. The synchronous path runs only on the non-forced bailout, before any instrumentation is active, so the forwarder is never traced; the async spawn stays for the post-init `complete` point. spawnSync exists since Node 0.11.12 and the guardrails still target >=0.8, which predates the exit bug, so the async path remains the fallback there. Refs: nodejs/node#63085 * fix(guardrails): bound the synchronous forwarder so it cannot hang the exit The synchronous bailout path validates only that DD_TELEMETRY_FORWARDER_PATH exists, then blocks on spawnSync with no timeout. A forwarder or wrapper that wedges before exiting would hang the injected process before guardrails return and before the app can reach its own process.exit() — reintroducing the exact hard hang this change exists to remove. Bound the send with a 1s timeout and SIGKILL, and log ETIMEDOUT distinctly; the telemetry is best-effort, so a bad forwarder degrades to a dropped point instead of a stuck process. * test(guardrails): cover the synchronous forwarder error branches The spawnSync bailout path logs two failures that had no coverage: a non-ETIMEDOUT spawn error (a forwarder that exists but cannot execute, e.g. EACCES) and a non-zero forwarder exit code. Both are reachable and mirror the async path's handling, so they get real cases beside the existing ETIMEDOUT test using the same mocked-spawnSync pattern.
…g exit (#9268) * fix(guardrails): forward abort telemetry synchronously to avoid a hung exit Single-step install injects the tracer via NODE_OPTIONS. When it bails out on an incompatible runtime it fires abort telemetry by spawning the forwarder with child_process.spawn, then the injected app returns and typically calls process.exit(). On Node 24.0.0/24.1.x a child still tearing down its stdio pipes deadlocks process.exit() (fixed upstream in 24.2), so the forwarder keeps the short-lived process alive until it is killed externally. Because injection is automatic, an unsupported-runtime customer with short-lived processes gets hangs. Forward the abort telemetry with spawnSync so the child is fully reaped before we return and nothing outlives the call to race the exit. The synchronous path runs only on the non-forced bailout, before any instrumentation is active, so the forwarder is never traced; the async spawn stays for the post-init `complete` point. spawnSync exists since Node 0.11.12 and the guardrails still target >=0.8, which predates the exit bug, so the async path remains the fallback there. Refs: nodejs/node#63085 * fix(guardrails): bound the synchronous forwarder so it cannot hang the exit The synchronous bailout path validates only that DD_TELEMETRY_FORWARDER_PATH exists, then blocks on spawnSync with no timeout. A forwarder or wrapper that wedges before exiting would hang the injected process before guardrails return and before the app can reach its own process.exit() — reintroducing the exact hard hang this change exists to remove. Bound the send with a 1s timeout and SIGKILL, and log ETIMEDOUT distinctly; the telemetry is best-effort, so a bad forwarder degrades to a dropped point instead of a stuck process. * test(guardrails): cover the synchronous forwarder error branches The spawnSync bailout path logs two failures that had no coverage: a non-ETIMEDOUT spawn error (a forwarder that exists but cannot execute, e.g. EACCES) and a non-zero forwarder exit code. Both are reachable and mirror the async path's handling, so they get real cases beside the existing ETIMEDOUT test using the same mocked-spawnSync pattern.
Summary
Single-step install (SSI) injects the tracer via
NODE_OPTIONS. On the abort path — an incompatible runtime, where the tracer does not initialize — the guardrails fire abort telemetry by spawning the forwarder withchild_process.spawn, then the injected app returns and typically callsprocess.exit(). On Node 24.0.0/24.1.x a child that is still tearing down its stdio pipes deadlocksprocess.exit()(fixed upstream in 24.2), so the forwarder keeps the short-lived process alive until it is killed externally.Because injection is automatic, this hangs a customer's short-lived processes whenever they run SSI on an unsupported Node 24.0.0/24.1 runtime — reproduced without any test harness (~70% of runs in a faithful sandbox; disabling the forwarder drops it to 0).
Forward the abort telemetry with
spawnSyncso the child is fully reaped before we return and nothing outlives the call to race the exit. The synchronous path runs only on the non-forced bailout, before any instrumentation is active, so the forwarder is never traced; the asyncspawnstays for the post-initcompletepoint.Why
spawnSyncexists since Node 0.11.12 and the guardrails still target>=0.8, which predates the exit bug, so the asyncspawnremains the fallback there. The functions are captured at module load, before the tracer wrapschild_process, so a forced init that runs afterwards cannot route the forwarder through the tracer's own instrumentation.This addresses the dominant, product-side cause. A small residual hang remains on Node 24.0.0/24.1 that is the runtime's own
process.exit()teardown bug (unreachable from our code, fixed in Node 24.2); the CI guardrail flake it still produces is handled separately.Refs: nodejs/node#63085