Skip to content

fix(crashtracking): resolve frames out-of-process on Linux#9237

Merged
BridgeAR merged 2 commits into
masterfrom
bengl/fix-crashtracker-linux-symbolication
Jul 7, 2026
Merged

fix(crashtracking): resolve frames out-of-process on Linux#9237
BridgeAR merged 2 commits into
masterfrom
bengl/fix-crashtracker-linux-symbolication

Conversation

@bengl

@bengl bengl commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Fixes crash-report symbolication mode selection on Linux. The resolve
mode was chosen with require('os').platform === 'linux' — comparing the
platform function reference to a string, which is always false.
As a result Linux always used EnabledWithInprocessSymbols instead of
the intended EnabledWithSymbolsInReceiver (out-of-process symbolication,
which the adjacent comment notes is the only supported path on Linux).

The fix imports platform from node:os and calls platform() === 'linux'.
A unit test now asserts resolve_frames matches the running platform, so
the regression is caught on Linux CI.

Motivation

Correctness fix found while investigating flaky SSI crashtracking system
tests. On Linux, crash reports should be symbolicated out-of-process by
the receiver; the buggy comparison silently forced in-process
symbolication instead. Latent since #7525 (2026-02-23).

Additional Notes

  • Scope: this is a symbolication-mode correctness fix. It does not
    fix the flaky test_docker_ssi_crash…test_crash[js-app] /
    test_telemetry[js-app] / test_install_supported_runtime[js-app]
    system tests. Investigation showed those failures are clustered at the
    pipeline level — an entire DOCKER_SSI stage intermittently reports no
    data to the test-agent across all Node versions and both arches, and the
    same pipelines also fail an asm-js-owned test. That is a scenario/CI
    infrastructure flake, not a dd-trace-js code defect, and is tracked
    separately.
  • The in-process path still produces a crash report (the test only asserts
    a report exists), so this bug affects symbol quality on Linux, not
    whether a report is generated.
  • Verified: mocha crashtracker.spec.js -> 16 passing; the new test fails
    against the old code on Linux. End-to-end Linux/Docker crash
    symbolication was not exercised locally.

The Linux/non-Linux resolve mode selection compared `os.platform`, the
function reference, against the string 'linux'. That comparison is
always false, so Linux always fell through to
'EnabledWithInprocessSymbols' instead of the intended
'EnabledWithSymbolsInReceiver'. Out-of-process symbolication is the
only supported path on Linux (per the adjacent comment and the
crashtracking team's original guidance), so in-process symbolication
during a live crash could leave the crash report unwritten.

Call `platform()` so the real platform is evaluated. Add a unit test
asserting resolve_frames matches the running platform, which fails
against the old code on Linux.
@dd-octo-sts

dd-octo-sts Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.6 MB
Deduped: 7.26 MB
No deduping: 7.26 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 437.94 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@bengl
bengl marked this pull request as ready for review July 7, 2026 14:53
@bengl
bengl requested a review from a team as a code owner July 7, 2026 14:53
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 7, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f664043 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 7, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-07 17:35:59

Comparing candidate commit f664043 in PR branch bengl/fix-crashtracker-linux-symbolication with baseline commit 8df3778 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2259 metrics, 27 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-177676.255µs; +177520.122µs] or [-6.755%; +6.749%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-222.258ms; +193.543ms] or [-8.875%; +7.728%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-156.802ms; +154.303ms] or [-5.447%; +5.360%]

scenario:appsec-control-20

  • unstable execution_time [-108.088ms; +116.063ms] or [-6.662%; +7.153%]

scenario:appsec-control-24

  • unstable execution_time [-103135.927µs; +104045.893µs] or [-8.489%; +8.564%]

scenario:appsec-control-26

  • unstable execution_time [-117036.529µs; +115285.162µs] or [-9.645%; +9.501%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-2.415s; +0.226s] or [-27.249%; +2.554%]
  • unstable execution_time [-2.452s; +0.245s] or [-25.627%; +2.559%]
  • unstable instructions [-20.5G instructions; +1.8G instructions] or [-28.187%; +2.449%]
  • unstable throughput [-77.888op/s; +738.338op/s] or [-2.244%; +21.271%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-1903.814ms; +595.532ms] or [-20.330%; +6.359%]
  • unstable execution_time [-1926.193ms; +619.990ms] or [-19.134%; +6.159%]
  • unstable instructions [-17.0G instructions; +4.9G instructions] or [-21.880%; +6.289%]
  • unstable throughput [-132.779op/s; +419.028op/s] or [-4.062%; +12.818%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-2571.429ms; +2155.437ms] or [-28.760%; +24.107%]
  • unstable execution_time [-2605.292ms; +2197.689ms] or [-27.060%; +22.827%]
  • unstable instructions [-21.7G instructions; +18.2G instructions] or [-29.722%; +24.924%]
  • unstable max_rss_usage [-8161.325KB; +8757.325KB] or [-5.127%; +5.501%]
  • unstable throughput [-605.617op/s; +789.902op/s] or [-17.559%; +22.902%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-3.244s; +0.425s] or [-32.047%; +4.198%]
  • unstable execution_time [-3.251s; +0.437s] or [-30.080%; +4.047%]
  • unstable instructions [-28.8G instructions; +3.9G instructions] or [-34.036%; +4.606%]
  • unstable throughput [-88.776op/s; +678.481op/s] or [-2.846%; +21.750%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-148.248ms; +550.095ms] or [-3.112%; +11.547%]
  • unstable execution_time [-149.525ms; +549.707ms] or [-3.091%; +11.362%]
  • unstable throughput [-188323.653op/s; +51224.205op/s] or [-10.853%; +2.952%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-7915.281KB; +6156.995KB] or [-5.987%; +4.657%]

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.52%. Comparing base (8df3778) to head (f664043).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #9237   +/-   ##
=======================================
  Coverage   96.52%   96.52%           
=======================================
  Files         905      905           
  Lines      119282   119282           
  Branches    20576    20558   -18     
=======================================
+ Hits       115132   115136    +4     
+ Misses       4150     4146    -4     
Flag Coverage Δ
aiguard 53.54% <100.00%> (-0.05%) ⬇️
aiguard-integration 57.37% <100.00%> (ø)
apm-bucket-0 53.76% <100.00%> (-0.05%) ⬇️
apm-bucket-1 58.08% <100.00%> (-0.05%) ⬇️
apm-bucket-2 56.10% <100.00%> (-0.05%) ⬇️
apm-capabilities-tracing 61.00% <100.00%> (ø)
apm-integrations-aerospike 52.92% <100.00%> (-0.05%) ⬇️
apm-integrations-confluentinc-kafka-javascript 57.05% <100.00%> (+0.01%) ⬆️
apm-integrations-couchbase 53.23% <100.00%> (-0.05%) ⬇️
apm-integrations-http 57.97% <100.00%> (-0.05%) ⬇️
apm-integrations-kafkajs 57.72% <100.00%> (-0.03%) ⬇️
apm-integrations-next 54.21% <100.00%> (-0.05%) ⬇️
apm-integrations-prisma 54.17% <100.00%> (-0.05%) ⬇️
apm-integrations-tedious 53.01% <100.00%> (-0.05%) ⬇️
appsec 69.03% <100.00%> (-0.05%) ⬇️
appsec-express_fastify_graphql 65.67% <100.00%> (-0.03%) ⬇️
appsec-integration 45.82% <100.00%> (ø)
appsec-kafka_ldapjs_lodash 59.11% <100.00%> (-0.04%) ⬇️
appsec-mongodb-core_mongoose_mysql 62.30% <100.00%> (-0.04%) ⬇️
appsec-next 52.84% <100.00%> (-0.04%) ⬇️
appsec-node-serialize_passport_postgres 61.91% <100.00%> (-0.04%) ⬇️
appsec-sourcing_stripe_template 60.25% <100.00%> (-0.04%) ⬇️
debugger 65.69% <100.00%> (ø)
instrumentations-bucket-0 48.74% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-1 54.95% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-10 56.62% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-11 48.70% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-12 48.81% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-13 48.64% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-2 49.71% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-3 53.81% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-4 49.13% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-5 52.99% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-6 55.60% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-7 53.59% <100.00%> (-0.06%) ⬇️
instrumentations-bucket-8 54.62% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-9 56.08% <100.00%> (-0.04%) ⬇️
instrumentations-instrumentation-couchbase 48.13% <100.00%> (-0.05%) ⬇️
instrumentations-integration-esbuild 34.20% <ø> (ø)
llmobs-ai_anthropic_bedrock 57.26% <100.00%> (-0.04%) ⬇️
llmobs-google-genai_langchain_vertex-ai 55.76% <100.00%> (-0.04%) ⬇️
llmobs-openai 57.56% <100.00%> (-0.04%) ⬇️
llmobs-sdk 59.97% <100.00%> (-0.05%) ⬇️
master-coverage 96.52% <100.00%> (?)
openfeature 54.59% <100.00%> (+<0.01%) ⬆️
openfeature-unit 49.79% <100.00%> (-0.05%) ⬇️
platform-core_esbuild_instrumentations-misc 38.66% <100.00%> (-0.03%) ⬇️
platform-integration 62.17% <100.00%> (ø)
platform-shimmer_unit-guardrails_webpack 37.45% <100.00%> (-0.03%) ⬇️
plugins-bucket-0 53.16% <100.00%> (-0.05%) ⬇️
plugins-bucket-1 55.05% <100.00%> (ø)
plugins-bucket-11 56.72% <100.00%> (-0.05%) ⬇️
plugins-bucket-17 56.73% <100.00%> (-0.05%) ⬇️
plugins-bucket-18 57.55% <100.00%> (-0.05%) ⬇️
plugins-bucket-19 57.07% <100.00%> (-0.05%) ⬇️
plugins-bucket-20 59.37% <100.00%> (-0.05%) ⬇️
plugins-bucket-4 53.83% <100.00%> (-0.05%) ⬇️
plugins-bullmq_cassandra_cookie 57.38% <100.00%> (-0.05%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 52.29% <100.00%> (-0.05%) ⬇️
plugins-fetch_fs_generic-pool 54.44% <100.00%> (-0.05%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 59.87% <100.00%> (-0.05%) ⬇️
plugins-hapi_hono_ioredis 55.85% <100.00%> (-0.05%) ⬇️
plugins-jest_knex_langgraph 51.64% <100.00%> (-0.05%) ⬇️
plugins-ldapjs_light-my-request_limitd-client 53.39% <100.00%> (-0.08%) ⬇️
plugins-lodash_mariadb_memcached 54.19% <100.00%> (-0.05%) ⬇️
plugins-mongodb_mongodb-core_mongoose 55.10% <100.00%> (-0.05%) ⬇️
plugins-multer_mysql_mysql2 54.35% <100.00%> (-0.05%) ⬇️
plugins-nats_node-serialize_opensearch 56.05% <100.00%> (-0.05%) ⬇️
plugins-passport-http_pino_postgres 54.72% <100.00%> (-0.05%) ⬇️
plugins-process_pug_redis 53.85% <100.00%> (-0.05%) ⬇️
plugins-undici_url_valkey 54.37% <100.00%> (-0.04%) ⬇️
plugins-vm_winston_ws 55.25% <100.00%> (-0.05%) ⬇️
profiling 58.19% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-aws-sdk 50.63% <100.00%> (-0.04%) ⬇️
serverless-aws-sdk-bedrockruntime 50.65% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-client 51.93% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-dynamodb 51.72% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-eventbridge 46.18% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-kinesis 54.68% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-lambda 52.76% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-s3 51.50% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-serverless-peer-service 54.71% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-sns 55.46% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-sqs 55.89% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-stepfunctions 51.22% <100.00%> (-0.05%) ⬇️
serverless-aws-sdk-util 48.42% <100.00%> (-0.05%) ⬇️
serverless-bucket-0 55.21% <100.00%> (ø)
serverless-bucket-1 56.00% <100.00%> (-0.05%) ⬇️
test-optimization-cucumber 72.92% <100.00%> (-0.04%) ⬇️
test-optimization-cypress 66.40% <100.00%> (+0.01%) ⬆️
test-optimization-jest 73.89% <100.00%> (-0.05%) ⬇️
test-optimization-mocha 74.30% <100.00%> (+0.04%) ⬆️
test-optimization-playwright-playwright-atr 61.57% <100.00%> (-0.01%) ⬇️
test-optimization-playwright-playwright-efd 61.76% <100.00%> (ø)
test-optimization-playwright-playwright-final-status 61.71% <100.00%> (-0.01%) ⬇️
test-optimization-playwright-playwright-impacted-tests 61.43% <100.00%> (+0.16%) ⬆️
test-optimization-playwright-playwright-reporting 61.58% <100.00%> (+0.05%) ⬆️
test-optimization-playwright-playwright-test-management 62.25% <100.00%> (-0.12%) ⬇️
test-optimization-playwright-playwright-test-span 61.48% <100.00%> (-0.06%) ⬇️
test-optimization-selenium 60.68% <100.00%> (-0.11%) ⬇️
test-optimization-testopt 59.13% <100.00%> (+0.08%) ⬆️
test-optimization-vitest 72.74% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BridgeAR
BridgeAR enabled auto-merge (squash) July 7, 2026 15:12
@BridgeAR
BridgeAR merged commit 0175f0c into master Jul 7, 2026
805 of 806 checks passed
@BridgeAR
BridgeAR deleted the bengl/fix-crashtracker-linux-symbolication branch July 7, 2026 17:38
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 7, 2026
The Linux/non-Linux resolve mode selection compared `os.platform`, the
function reference, against the string 'linux'. That comparison is
always false, so Linux always fell through to
'EnabledWithInprocessSymbols' instead of the intended
'EnabledWithSymbolsInReceiver'. Out-of-process symbolication is the
only supported path on Linux (per the adjacent comment and the
crashtracking team's original guidance), so in-process symbolication
during a live crash could leave the crash report unwritten.

Call `platform()` so the real platform is evaluated. Add a unit test
asserting resolve_frames matches the running platform, which fails
against the old code on Linux.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 7, 2026
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 7, 2026
The Linux/non-Linux resolve mode selection compared `os.platform`, the
function reference, against the string 'linux'. That comparison is
always false, so Linux always fell through to
'EnabledWithInprocessSymbols' instead of the intended
'EnabledWithSymbolsInReceiver'. Out-of-process symbolication is the
only supported path on Linux (per the adjacent comment and the
crashtracking team's original guidance), so in-process symbolication
during a live crash could leave the crash report unwritten.

Call `platform()` so the real platform is evaluated. Add a unit test
asserting resolve_frames matches the running platform, which fails
against the old code on Linux.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 7, 2026
juan-fernandez pushed a commit that referenced this pull request Jul 8, 2026
The Linux/non-Linux resolve mode selection compared `os.platform`, the
function reference, against the string 'linux'. That comparison is
always false, so Linux always fell through to
'EnabledWithInprocessSymbols' instead of the intended
'EnabledWithSymbolsInReceiver'. Out-of-process symbolication is the
only supported path on Linux (per the adjacent comment and the
crashtracking team's original guidance), so in-process symbolication
during a live crash could leave the crash report unwritten.

Call `platform()` so the real platform is evaluated. Add a unit test
asserting resolve_frames matches the running platform, which fails
against the old code on Linux.
juan-fernandez pushed a commit that referenced this pull request Jul 8, 2026
The Linux/non-Linux resolve mode selection compared `os.platform`, the
function reference, against the string 'linux'. That comparison is
always false, so Linux always fell through to
'EnabledWithInprocessSymbols' instead of the intended
'EnabledWithSymbolsInReceiver'. Out-of-process symbolication is the
only supported path on Linux (per the adjacent comment and the
crashtracking team's original guidance), so in-process symbolication
during a live crash could leave the crash report unwritten.

Call `platform()` so the real platform is evaluated. Add a unit test
asserting resolve_frames matches the running platform, which fails
against the old code on Linux.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants