[DI] Workaround bug in AsyncLocalStorage which would otherwise throw by watson · Pull Request #5290 · DataDog/dd-trace-js

watson · 2025-02-18T12:59:11Z

What does this PR do?

This is a workaround for, what seems like a bug in Node.js core, that seems to trigger when, among other things, a lot of timers are being created very rapidly. This makes the call to setTimeout throw an error from within AsyncLocalStorage._propagate with the following TypeError:

Cannot read properties of undefined (reading 'Symbol(kResourceStore)')

Source of the bug: https://github.com/nodejs/node/blob/v18.20.6/lib/async_hooks.js#L312

Motivation

Don't crash customer applications.

Plugin Checklist

Additional Notes

watson · 2025-02-18T12:59:32Z

[DI] Workaround bug in AsyncLocalStorage which would otherwise throw #5290 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2025-02-18T13:00:29Z

Overall package size

Self size: 8.75 MB
Deduped: 94.95 MB
No deduping: 95.47 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.5.1 | 9.79 MB | 10.17 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 835.4 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2025-02-18T13:00:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.90%. Comparing base (172bc66) to head (22ab9c4).
Report is 272 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5290      +/-   ##
==========================================
+ Coverage   80.80%   80.90%   +0.09%     
==========================================
  Files         487      487              
  Lines       21710    21797      +87     
==========================================
+ Hits        17542    17634      +92     
+ Misses       4168     4163       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-datadog-prod-us1 · 2025-02-18T13:06:06Z

Datadog Report

Branch report: watson/DEBUG-3526/nodejs-core-bug-workaround
Commit report: bd7cd0a
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 674 Passed, 0 Skipped, 16m 8.67s Total Time

pr-commenter · 2025-02-18T13:08:51Z

Benchmarks

Benchmark execution time: 2025-02-19 09:23:56

Comparing candidate commit 22ab9c4 in PR branch watson/DEBUG-3526/nodejs-core-bug-workaround with baseline commit 172bc66 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 915 metrics, 18 unstable metrics.

cbeauchesne

What about adding failing system-tests scenarios?

it's here : https://github.com/DataDog/dd-trace-js/blob/master/.github/workflows/system-tests.yml#L34

Just add a line : scenarios: DEBUGGER_PII_REDACTION, DEBUGGER_PROBES_SNAPSHOT

This is a workaround for, what seems like a bug in Node.js core, that seems to trigger when, among other things, a lot of timers are being created very rapidly. This makes the call to `setTimeout` throw an error from within `AsyncLocalStorage._propagate` with the following TypeError: Cannot read properties of undefined (reading 'Symbol(kResourceStore)')

…5290) This is a workaround for, what seems like a bug in Node.js core, that seems to trigger when, among other things, a lot of timers are being created very rapidly. This makes the call to `setTimeout` throw an error from within `AsyncLocalStorage._propagate` with the following TypeError: Cannot read properties of undefined (reading 'Symbol(kResourceStore)')

watson requested review from a team as code owners February 18, 2025 12:59

watson self-assigned this Feb 18, 2025

watson added semver-patch debugger Dynamic Instrumentation & Live Debugger labels Feb 18, 2025

cbeauchesne reviewed Feb 18, 2025

View reviewed changes

watson force-pushed the watson/DEBUG-3526/nodejs-core-bug-workaround branch from 70ceee5 to 4d83e30 Compare February 18, 2025 15:13

watson requested a review from a team as a code owner February 18, 2025 15:13

watson added 3 commits February 19, 2025 06:40

Run all system tests to verify if this actually fixes the bug

ab0f733

Compromise

75e555e

watson force-pushed the watson/DEBUG-3526/nodejs-core-bug-workaround branch from 4d83e30 to 75e555e Compare February 19, 2025 09:00

watson requested a review from cbeauchesne February 19, 2025 09:00

cbeauchesne previously approved these changes Feb 19, 2025

View reviewed changes

simon-id reviewed Feb 19, 2025

View reviewed changes

Comment thread packages/dd-trace/src/debugger/devtools_client/source-maps.js Outdated

Address review comments

22ab9c4

watson dismissed cbeauchesne’s stale review via 22ab9c4 February 19, 2025 09:14

simon-id approved these changes Feb 19, 2025

View reviewed changes

watson enabled auto-merge (squash) February 19, 2025 10:14

watson merged commit 77abd1e into master Feb 19, 2025

watson deleted the watson/DEBUG-3526/nodejs-core-bug-workaround branch February 19, 2025 10:17

watson mentioned this pull request Feb 19, 2025

v5.37.1 proposal #5295

Merged

watson mentioned this pull request Feb 20, 2025

[DI] Remove source map cache #5302

Merged

6 tasks

watson added the bug Something isn't working label May 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DI] Workaround bug in AsyncLocalStorage which would otherwise throw#5290

[DI] Workaround bug in AsyncLocalStorage which would otherwise throw#5290
watson merged 4 commits into
masterfrom
watson/DEBUG-3526/nodejs-core-bug-workaround

watson commented Feb 18, 2025

Uh oh!

watson commented Feb 18, 2025

Uh oh!

github-actions Bot commented Feb 18, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Feb 18, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Feb 18, 2025 •

edited

Loading

Uh oh!

pr-commenter Bot commented Feb 18, 2025 •

edited

Loading

Uh oh!

cbeauchesne left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

watson commented Feb 18, 2025

What does this PR do?

Motivation

Plugin Checklist

Additional Notes

Uh oh!

watson commented Feb 18, 2025

Uh oh!

github-actions Bot commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

codecov Bot commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-datadog-prod-us1 Bot commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

pr-commenter Bot commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Feb 18, 2025 •

edited

Loading

codecov Bot commented Feb 18, 2025 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Feb 18, 2025 •

edited

Loading

pr-commenter Bot commented Feb 18, 2025 •

edited

Loading