Skip to content

ci(scripts): disable V8 Maglev for Windows test children#9131

Merged
pabloerhard merged 2 commits into
masterfrom
BridgeAR/2026-06-29-windows-maglev-exit-crash
Jun 30, 2026
Merged

ci(scripts): disable V8 Maglev for Windows test children#9131
pabloerhard merged 2 commits into
masterfrom
BridgeAR/2026-06-29-windows-maglev-exit-crash

Conversation

@BridgeAR

Copy link
Copy Markdown
Member

Summary

Windows test jobs intermittently fail with a crashed spec file and exit code
3221226505 (0xC0000409 / STATUS_STACK_BUFFER_OVERRUN) — no stderr, no
crash report, no mocha failure. scripts/mocha-run-file.js forces
process.exit() as soon as mocha finishes, and on Windows that races V8's
Maglev teardown while libuv still has in-flight sockets from a spec's real HTTP
traffic. The runner only sees a non-zero child exit and reports the file as
crashed. The file in the originating run was inferred_proxy.spec.js, but it is
just whichever network-heavy spec lost the race — its plugin logic is sound.

scripts/mocha-parallel-files.js now injects --no-maglev into the spawned
per-file node processes on win32 only, sidestepping the faulty tier.

Why

--no-maglev is a V8 flag that cannot be set through NODE_OPTIONS, so it has
to be a direct CLI argument on each spawned child — the parent runner can't
forward it any other way. The crash is upstream (nodejs/node#62260, still open);
this is a stop-gap until that lands, gated to Windows where the abort occurs.

Test plan

  • Windows test jobs (tracing-windows, and other Windows suites routed
    through mocha-parallel-files.js) stay green across reruns.
  • Non-Windows path unchanged: node scripts/mocha-parallel-files.js --timeout 30000 -- "packages/dd-trace/test/plugins/util/inferred_proxy.spec.js" passes locally with no flag added.

Refs: nodejs/node#62260

Network-heavy specs intermittently abort with STATUS_STACK_BUFFER_OVERRUN
(0xC0000409) on Windows: mocha-run-file forces process.exit() the moment
mocha finishes, and that races V8's Maglev teardown while libuv still has
in-flight sockets from the spec's real HTTP traffic. The child dies with no
stderr and no crash report, so mocha-parallel-files only sees a non-zero exit
and reports the file as crashed (e.g. inferred_proxy.spec.js, which is just
whichever network spec lost the race that run).

--no-maglev sidesteps the faulty tier and can only be passed as a CLI flag,
not through NODE_OPTIONS, so the runner injects it into the spawned per-file
node processes on win32.

Refs: nodejs/node#62260
@dd-octo-sts

dd-octo-sts Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.42 MB
Deduped: 7.48 MB
No deduping: 7.48 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 29, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 88.01% (+0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4540f29 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 29, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-29 15:19:38

Comparing candidate commit 4540f29 in PR branch BridgeAR/2026-06-29-windows-maglev-exit-crash with baseline commit 243c792 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2255 metrics, 31 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-180769.919µs; +182290.019µs] or [-6.840%; +6.897%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-210.022ms; +212.095ms] or [-8.336%; +8.418%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-158987.043µs; +158143.043µs] or [-5.517%; +5.488%]

scenario:appsec-control-20

  • unstable execution_time [-167.727ms; +158.415ms] or [-9.575%; +9.043%]

scenario:appsec-control-24

  • unstable execution_time [-103777.312µs; +102666.245µs] or [-8.519%; +8.428%]

scenario:appsec-control-26

  • unstable execution_time [-101.500ms; +117.687ms] or [-8.453%; +9.801%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1909.776ms; +3029.979ms] or [-24.067%; +38.184%]
  • unstable execution_time [-1917.589ms; +3047.725ms] or [-22.234%; +35.338%]
  • unstable instructions [-16.3G instructions; +26.0G instructions] or [-25.286%; +40.329%]
  • unstable max_rss_usage [-6.838MB; +11.099MB] or [-4.330%; +7.027%]
  • unstable throughput [-892.816op/s; +563.255op/s] or [-23.327%; +14.716%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-3789.099ms; +2532.773ms] or [-36.114%; +24.140%]
  • unstable execution_time [-3870.986ms; +2628.602ms] or [-34.580%; +23.482%]
  • unstable instructions [-33.2G instructions; +22.2G instructions] or [-37.738%; +25.269%]
  • unstable max_rss_usage [-11.561MB; +7.148MB] or [-7.083%; +4.379%]
  • unstable throughput [-575.306op/s; +862.431op/s] or [-18.917%; +28.358%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-1653.288ms; +529.587ms] or [-20.879%; +6.688%]
  • unstable execution_time [-1660.177ms; +539.521ms] or [-19.355%; +6.290%]
  • unstable instructions [-13.8G instructions; +4.5G instructions] or [-21.520%; +7.020%]
  • unstable throughput [-159.906op/s; +492.179op/s] or [-4.156%; +12.792%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-2587.871ms; +2517.352ms] or [-30.633%; +29.799%]
  • unstable execution_time [-2639.375ms; +2592.802ms] or [-28.877%; +28.368%]
  • unstable instructions [-22093.9M instructions; +22049.3M instructions] or [-32.031%; +31.967%]
  • unstable max_rss_usage [-9503.534KB; +9226.734KB] or [-5.940%; +5.767%]
  • unstable throughput [-765.998op/s; +791.632op/s] or [-20.841%; +21.538%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-244.076ms; +492.560ms] or [-5.037%; +10.165%]
  • unstable execution_time [-246.720ms; +487.615ms] or [-5.013%; +9.907%]
  • unstable throughput [-166100.868op/s; +85729.844op/s] or [-9.729%; +5.022%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-23716.041KB; +23545.641KB] or [-6.344%; +6.298%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable max_rss_usage [-87.893MB; +43.959MB] or [-37.286%; +18.648%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-3.273MB; +6.790MB] or [-4.121%; +8.550%]

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.71%. Comparing base (243c792) to head (4540f29).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9131      +/-   ##
==========================================
+ Coverage   93.69%   93.71%   +0.02%     
==========================================
  Files         889      891       +2     
  Lines       50861    51161     +300     
  Branches    11832    11900      +68     
==========================================
+ Hits        47653    47948     +295     
- Misses       3208     3213       +5     
Flag Coverage Δ
aiguard 34.88% <ø> (-0.08%) ⬇️
aiguard-integration 41.79% <ø> (ø)
apm-bucket-0 34.85% <ø> (-0.08%) ⬇️
apm-bucket-1 40.34% <ø> (-0.07%) ⬇️
apm-bucket-2 37.35% <ø> (-0.07%) ⬇️
apm-capabilities-tracing 48.75% <ø> (+0.45%) ⬆️
apm-integrations-aerospike 33.13% <ø> (-0.08%) ⬇️
apm-integrations-confluentinc-kafka-javascript 40.04% <ø> (-0.08%) ⬇️
apm-integrations-couchbase 33.41% <ø> (-0.08%) ⬇️
apm-integrations-http 42.04% <ø> (-0.07%) ⬇️
apm-integrations-kafkajs 40.27% <ø> (-0.08%) ⬇️
apm-integrations-next 29.51% <ø> (-0.04%) ⬇️
apm-integrations-prisma 35.04% <ø> (-0.07%) ⬇️
apm-integrations-tedious 33.90% <ø> (-0.08%) ⬇️
appsec 57.38% <ø> (+0.01%) ⬆️
appsec-express_fastify_graphql 53.78% <ø> (-0.03%) ⬇️
appsec-integration 36.25% <ø> (ø)
appsec-kafka_ldapjs_lodash 43.68% <ø> (+0.02%) ⬆️
appsec-mongodb-core_mongoose_mysql 48.81% <ø> (-0.06%) ⬇️
appsec-next 28.01% <ø> (-0.07%) ⬇️
appsec-node-serialize_passport_postgres 48.01% <ø> (-0.06%) ⬇️
appsec-sourcing_stripe_template 45.56% <ø> (-0.06%) ⬇️
debugger 44.37% <ø> (-0.05%) ⬇️
instrumentations-bucket-0 28.16% <ø> (-0.08%) ⬇️
instrumentations-bucket-1 37.41% <ø> (-0.07%) ⬇️
instrumentations-bucket-10 40.44% <ø> (-0.07%) ⬇️
instrumentations-bucket-11 27.96% <ø> (-0.08%) ⬇️
instrumentations-bucket-12 28.67% <ø> (-0.08%) ⬇️
instrumentations-bucket-13 27.79% <ø> (-0.08%) ⬇️
instrumentations-bucket-2 30.24% <ø> (-0.07%) ⬇️
instrumentations-bucket-3 35.91% <ø> (-0.07%) ⬇️
instrumentations-bucket-4 28.07% <ø> (-0.08%) ⬇️
instrumentations-bucket-5 36.28% <ø> (-0.07%) ⬇️
instrumentations-bucket-6 38.26% <ø> (-0.08%) ⬇️
instrumentations-bucket-7 36.01% <ø> (-0.07%) ⬇️
instrumentations-bucket-8 36.96% <ø> (-0.07%) ⬇️
instrumentations-bucket-9 39.53% <ø> (-0.07%) ⬇️
instrumentations-instrumentation-couchbase 46.54% <ø> (ø)
instrumentations-integration-esbuild 24.87% <ø> (ø)
llmobs-ai_anthropic_bedrock 39.54% <ø> (-0.07%) ⬇️
llmobs-google-genai_langchain_vertex-ai 36.98% <ø> (-0.06%) ⬇️
llmobs-openai 39.59% <ø> (-0.07%) ⬇️
llmobs-sdk 43.60% <ø> (-0.08%) ⬇️
master-coverage 93.71% <ø> (?)
openfeature 37.76% <ø> (ø)
openfeature-unit 50.39% <ø> (ø)
platform-core_esbuild_instrumentations-misc 23.36% <ø> (ø)
platform-integration 47.43% <ø> (ø)
platform-shimmer_unit-guardrails_webpack 18.89% <ø> (ø)
plugins-bucket-0 36.34% <ø> (-0.07%) ⬇️
plugins-bucket-1 39.61% <ø> (ø)
plugins-bucket-11 38.47% <ø> (-0.08%) ⬇️
plugins-bucket-17 39.16% <ø> (-0.07%) ⬇️
plugins-bucket-18 42.01% <ø> (-0.07%) ⬇️
plugins-bucket-19 39.57% <ø> (-0.07%) ⬇️
plugins-bucket-20 43.26% <ø> (-0.07%) ⬇️
plugins-bucket-4 37.72% <ø> (-0.07%) ⬇️
plugins-bullmq_cassandra_cookie 39.77% <ø> (-0.07%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 33.23% <ø> (-0.08%) ⬇️
plugins-fetch_fs_generic-pool 36.06% <ø> (-0.07%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 43.11% <ø> (-0.07%) ⬇️
plugins-hapi_hono_ioredis 37.79% <ø> (-0.07%) ⬇️
plugins-jest_knex_langgraph 32.61% <ø> (-0.08%) ⬇️
plugins-ldapjs_light-my-request_limitd-client 27.92% <ø> (-0.02%) ⬇️
plugins-lodash_mariadb_memcached 35.18% <ø> (-0.08%) ⬇️
plugins-mongodb_mongodb-core_mongoose 36.32% <ø> (-0.07%) ⬇️
plugins-multer_mysql_mysql2 35.15% <ø> (-0.08%) ⬇️
plugins-nats_node-serialize_opensearch 37.16% <ø> (-0.08%) ⬇️
plugins-passport-http_pino_postgres 35.54% <ø> (-0.08%) ⬇️
plugins-process_pug_redis 34.27% <ø> (-0.08%) ⬇️
plugins-undici_url_valkey 35.89% <ø> (-0.07%) ⬇️
plugins-vm_winston_ws 37.56% <ø> (-0.08%) ⬇️
profiling 43.62% <ø> (-0.07%) ⬇️
serverless-aws-sdk-aws-sdk 33.30% <ø> (-0.07%) ⬇️
serverless-aws-sdk-bedrockruntime 32.17% <ø> (-0.07%) ⬇️
serverless-aws-sdk-client 37.21% <ø> (ø)
serverless-aws-sdk-dynamodb 34.23% <ø> (+0.06%) ⬆️
serverless-aws-sdk-eventbridge 27.25% <ø> (-0.07%) ⬇️
serverless-aws-sdk-kinesis 37.41% <ø> (-0.07%) ⬇️
serverless-aws-sdk-lambda 34.60% <ø> (-0.07%) ⬇️
serverless-aws-sdk-s3 32.60% <ø> (-0.07%) ⬇️
serverless-aws-sdk-serverless-peer-service 39.47% <ø> (-0.08%) ⬇️
serverless-aws-sdk-sns 38.26% <ø> (-0.07%) ⬇️
serverless-aws-sdk-sqs 38.01% <ø> (-0.07%) ⬇️
serverless-aws-sdk-stepfunctions 33.19% <ø> (-0.07%) ⬇️
serverless-aws-sdk-util 47.95% <ø> (ø)
serverless-bucket-0 39.47% <ø> (ø)
serverless-lambda 34.33% <ø> (-0.09%) ⬇️
test-optimization-cucumber 52.32% <ø> (-0.14%) ⬇️
test-optimization-cypress 49.53% <ø> (-0.04%) ⬇️
test-optimization-jest 55.36% <ø> (-0.21%) ⬇️
test-optimization-mocha 53.43% <ø> (-0.18%) ⬇️
test-optimization-playwright-playwright-atr 43.33% <ø> (-0.17%) ⬇️
test-optimization-playwright-playwright-efd 43.62% <ø> (-0.17%) ⬇️
test-optimization-playwright-playwright-final-status 43.74% <ø> (-0.22%) ⬇️
test-optimization-playwright-playwright-impacted-tests 43.16% <ø> (-0.25%) ⬇️
test-optimization-playwright-playwright-reporting 43.57% <ø> (+0.15%) ⬆️
test-optimization-playwright-playwright-test-management 44.73% <ø> (-0.12%) ⬇️
test-optimization-playwright-playwright-test-span 44.52% <ø> (-0.19%) ⬇️
test-optimization-selenium 45.29% <ø> (-0.17%) ⬇️
test-optimization-testopt 48.04% <ø> (-0.16%) ⬇️
test-optimization-vitest 50.64% <ø> (-0.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BridgeAR
BridgeAR marked this pull request as ready for review June 29, 2026 11:52
@BridgeAR
BridgeAR requested a review from a team as a code owner June 29, 2026 11:52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 407dbd80ec

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread scripts/mocha-parallel-files.js Outdated
The Windows --no-maglev workaround was appended on every win32 child, but the
top-level --maglev/--no-maglev toggle only exists from V8 11 (Node 20). On the
supported Node 18 line (V8 10) the flag does not exist, so each spawned spec
aborts with `bad option: --no-maglev` (exit 9) before mocha runs, breaking the
parallel runner for Windows Node 18. Gate on the running V8 major, which the
children inherit via the shared binary.
@pabloerhard
pabloerhard merged commit 530ff3d into master Jun 30, 2026
805 checks passed
@pabloerhard
pabloerhard deleted the BridgeAR/2026-06-29-windows-maglev-exit-crash branch June 30, 2026 16:08
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 1, 2026
* ci(scripts): disable V8 Maglev for Windows test children

Network-heavy specs intermittently abort with STATUS_STACK_BUFFER_OVERRUN
(0xC0000409) on Windows: mocha-run-file forces process.exit() the moment
mocha finishes, and that races V8's Maglev teardown while libuv still has
in-flight sockets from the spec's real HTTP traffic. The child dies with no
stderr and no crash report, so mocha-parallel-files only sees a non-zero exit
and reports the file as crashed (e.g. inferred_proxy.spec.js, which is just
whichever network spec lost the race that run).

--no-maglev sidesteps the faulty tier and can only be passed as a CLI flag,
not through NODE_OPTIONS, so the runner injects it into the spawned per-file
node processes on win32.

Refs: nodejs/node#62260

* ci(scripts): gate --no-maglev by V8 version

The Windows --no-maglev workaround was appended on every win32 child, but the
top-level --maglev/--no-maglev toggle only exists from V8 11 (Node 20). On the
supported Node 18 line (V8 10) the flag does not exist, so each spawned spec
aborts with `bad option: --no-maglev` (exit 9) before mocha runs, breaking the
parallel runner for Windows Node 18. Gate on the running V8 major, which the
children inherit via the shared binary.
This was referenced Jul 1, 2026
rochdev pushed a commit that referenced this pull request Jul 2, 2026
* ci(scripts): disable V8 Maglev for Windows test children

Network-heavy specs intermittently abort with STATUS_STACK_BUFFER_OVERRUN
(0xC0000409) on Windows: mocha-run-file forces process.exit() the moment
mocha finishes, and that races V8's Maglev teardown while libuv still has
in-flight sockets from the spec's real HTTP traffic. The child dies with no
stderr and no crash report, so mocha-parallel-files only sees a non-zero exit
and reports the file as crashed (e.g. inferred_proxy.spec.js, which is just
whichever network spec lost the race that run).

--no-maglev sidesteps the faulty tier and can only be passed as a CLI flag,
not through NODE_OPTIONS, so the runner injects it into the spawned per-file
node processes on win32.

Refs: nodejs/node#62260

* ci(scripts): gate --no-maglev by V8 version

The Windows --no-maglev workaround was appended on every win32 child, but the
top-level --maglev/--no-maglev toggle only exists from V8 11 (Node 20). On the
supported Node 18 line (V8 10) the flag does not exist, so each spawned spec
aborts with `bad option: --no-maglev` (exit 9) before mocha runs, breaking the
parallel runner for Windows Node 18. Gate on the running V8 major, which the
children inherit via the shared binary.
rochdev pushed a commit that referenced this pull request Jul 2, 2026
* ci(scripts): disable V8 Maglev for Windows test children

Network-heavy specs intermittently abort with STATUS_STACK_BUFFER_OVERRUN
(0xC0000409) on Windows: mocha-run-file forces process.exit() the moment
mocha finishes, and that races V8's Maglev teardown while libuv still has
in-flight sockets from the spec's real HTTP traffic. The child dies with no
stderr and no crash report, so mocha-parallel-files only sees a non-zero exit
and reports the file as crashed (e.g. inferred_proxy.spec.js, which is just
whichever network spec lost the race that run).

--no-maglev sidesteps the faulty tier and can only be passed as a CLI flag,
not through NODE_OPTIONS, so the runner injects it into the spawned per-file
node processes on win32.

Refs: nodejs/node#62260

* ci(scripts): gate --no-maglev by V8 version

The Windows --no-maglev workaround was appended on every win32 child, but the
top-level --maglev/--no-maglev toggle only exists from V8 11 (Node 20). On the
supported Node 18 line (V8 10) the flag does not exist, so each spawned spec
aborts with `bad option: --no-maglev` (exit 9) before mocha runs, breaking the
parallel runner for Windows Node 18. Gate on the running V8 major, which the
children inherit via the shared binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants