fix(test-optimization): support cucumber 13.1 results by juan-fernandez · Pull Request #9370 · DataDog/dd-trace-js

juan-fernandez · 2026-07-15T08:57:22Z

What does this PR do?

Updates the Cucumber instrumentation for the runtime result introduced in Cucumber 13.1.0. The wrapper now reads the success field for 13.1+ while returning Cucumber's original result object unchanged.

Also bumps the pinned @cucumber/cucumber test dependency from 13.0.0 to 13.1.0.

Motivation

Cucumber 13.1 changed Coordinator.run() from returning a boolean to returning { success, exception? }. The instrumentation treated that object as a boolean, so failed test sessions were reported as passing.

This fixes the failures from the latest-version compatibility job.

Additional Notes

Validation:

Cucumber 13.1 regression set: 7 passing through the integration coverage harness.
Cucumber 13.0 compatibility check: 2 passing.
Targeted ESLint and git diff --check pass.

dd-octo-sts · 2026-07-15T08:58:10Z

Overall package size

Self size: 6.77 MB
Deduped: 7.42 MB
No deduping: 7.42 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

datadog-official · 2026-07-15T09:00:31Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 96.61% (-0.00%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 781b5d5 | Docs | Datadog PR Page | Give us feedback!}

datadog-official · 2026-07-15T09:01:15Z

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

datadog-official

More details

The PR correctly fixes a critical bug where Cucumber 13.1's new result format { success, exception? } was being treated as a boolean, causing failed tests to be reported as passing. The fix conditionally extracts the success field for Cucumber 13.1+ while maintaining backward compatibility with Cucumber 13.0. Comprehensive validation confirms the fix handles both versions correctly without regressions.

Was this helpful? React 👍 or 👎

_{📊 Validated against 5 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 781b5d5 · What is Autotest? · Any feedback? Reach out in #autotest}

pr-commenter · 2026-07-15T09:14:00Z

Benchmarks

Benchmark execution time: 2026-07-15 09:13:56

Comparing candidate commit 781b5d5 in PR branch juan-fernandez/fix-cucumber-13-1-result with baseline commit 66bce78 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2318 metrics, 40 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

unstable execution_time [-215598.558µs; +213786.658µs] or [-8.145%; +8.077%]

scenario:appsec-appsec-enabled-26

unstable execution_time [-230.110ms; +227.298ms] or [-9.084%; +8.973%]

scenario:appsec-appsec-enabled-with-attacks-24

unstable execution_time [-160.029ms; +156.172ms] or [-5.209%; +5.084%]

scenario:appsec-appsec-enabled-with-attacks-26

unstable execution_time [-184.829ms; +177.398ms] or [-6.395%; +6.138%]

scenario:appsec-control-20

unstable execution_time [-120.884ms; +131.132ms] or [-7.354%; +7.977%]

scenario:appsec-control-24

unstable execution_time [-98.790ms; +123.030ms] or [-8.079%; +10.061%]

scenario:appsec-control-26

unstable execution_time [-122.680ms; +125.902ms] or [-9.937%; +10.198%]

scenario:appsec-iast-no-vulnerability-control-20

unstable execution_time [-18.226ms; +23.698ms] or [-7.059%; +9.178%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

unstable execution_time [-12766.798µs; +13860.065µs] or [-4.918%; +5.339%]

scenario:appsec-iast-with-vulnerability-control-20

unstable execution_time [-25.149ms; +36.587ms] or [-4.583%; +6.668%]

scenario:debugger-line-probe-with-snapshot-default-24

unstable cpu_user_time [-3240.532ms; +2376.481ms] or [-34.398%; +25.226%]
unstable execution_time [-3385.774ms; +2500.298ms] or [-33.366%; +24.640%]
unstable instructions [-27.3G instructions; +19.8G instructions] or [-35.366%; +25.675%]
unstable max_rss_usage [-13.763MB; +9.712MB] or [-8.530%; +6.019%]
unstable throughput [-660.574op/s; +897.952op/s] or [-19.729%; +26.819%]

scenario:debugger-line-probe-with-snapshot-minimal-24

unstable cpu_user_time [-1673.176ms; +668.087ms] or [-20.078%; +8.017%]
unstable execution_time [-1664.789ms; +662.793ms] or [-18.431%; +7.338%]
unstable instructions [-14.3G instructions; +5.5G instructions] or [-20.969%; +8.118%]
unstable throughput [-204.871op/s; +448.140op/s] or [-5.625%; +12.303%]

scenario:debugger-line-probe-with-snapshot-minimal-26

unstable cpu_user_time [-2047.373ms; +3327.568ms] or [-21.888%; +35.575%]
unstable execution_time [-2018.902ms; +3336.516ms] or [-20.038%; +33.115%]
unstable instructions [-18.1G instructions; +29.5G instructions] or [-23.224%; +37.941%]
unstable max_rss_usage [-6.027MB; +10.443MB] or [-3.781%; +6.551%]
unstable throughput [-733.885op/s; +433.277op/s] or [-22.520%; +13.295%]

scenario:debugger-line-probe-without-snapshot-26

unstable cpu_user_time [-1896.015ms; +555.865ms] or [-20.295%; +5.950%]
unstable execution_time [-1904.664ms; +545.086ms] or [-18.944%; +5.421%]
unstable instructions [-16.7G instructions; +4.9G instructions] or [-21.404%; +6.329%]
unstable throughput [-110.881op/s; +424.179op/s] or [-3.394%; +12.983%]

scenario:dogstatsd-with-tags-20

unstable cpu_user_time [-248.324ms; +410.028ms] or [-5.022%; +8.292%]
unstable execution_time [-246.996ms; +418.419ms] or [-4.921%; +8.337%]
unstable throughput [-145331.865op/s; +82984.999op/s] or [-8.685%; +4.959%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-20

unstable cpu_user_time [-3424167.079ns; +3424167.079ns] or [-5.091%; +5.091%]
unstable throughput [-26800.588op/s; +26800.588op/s] or [-5.351%; +5.351%]

scenario:plugin-graphql-long-with-depth-on-max-20

unstable cpu_user_time [-713386.693µs; +713147.493µs] or [-5.560%; +5.558%]
unstable execution_time [-721362.370µs; +719460.070µs] or [-5.502%; +5.487%]
unstable throughput [-3.315op/s; +3.320op/s] or [-5.401%; +5.409%]

scenario:plugin-memcached-get-26

unstable cpu_user_time [-182.369ms; +302.143ms] or [-7.805%; +12.930%]
unstable execution_time [-187.656ms; +306.880ms] or [-7.801%; +12.758%]
unstable throughput [-1318018.766op/s; +831847.456op/s] or [-7.934%; +5.007%]

scenario:test-optimization-large-suite-20

unstable max_rss_usage [-3.352MB; +6.819MB] or [-4.236%; +8.616%]

codecov · 2026-07-15T09:16:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.65%. Comparing base (66bce78) to head (781b5d5).
⚠️ Report is 36 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9370      +/-   ##
==========================================
+ Coverage   96.57%   96.65%   +0.08%     
==========================================
  Files         918      923       +5     
  Lines      121577   122845    +1268     
  Branches    21074    21456     +382     
==========================================
+ Hits       117407   118730    +1323     
+ Misses       4170     4115      -55

Flag	Coverage Δ
aiguard	`53.65% <ø> (+0.04%)`	⬆️
aiguard-integration	`57.10% <ø> (+0.09%)`	⬆️
apm-bucket-0	`53.88% <ø> (+0.04%)`	⬆️
apm-bucket-1	`58.45% <ø> (+0.33%)`	⬆️
apm-bucket-2	`56.18% <ø> (+0.02%)`	⬆️
apm-capabilities-tracing	`60.62% <ø> (+0.30%)`	⬆️
apm-integrations-aerospike	`53.03% <ø> (+0.05%)`	⬆️
apm-integrations-confluentinc-kafka-javascript	`57.12% <ø> (+0.01%)`	⬆️
apm-integrations-couchbase	`53.37% <ø> (+0.05%)`	⬆️
apm-integrations-http	`58.04% <ø> (+0.03%)`	⬆️
apm-integrations-kafkajs	`57.85% <ø> (+0.09%)`	⬆️
apm-integrations-next	`54.26% <ø> (-0.03%)`	⬇️
apm-integrations-prisma	`54.26% <ø> (+0.04%)`	⬆️
apm-integrations-tedious	`53.14% <ø> (+0.04%)`	⬆️
appsec	`68.86% <ø> (-0.14%)`	⬇️
appsec-express_fastify_graphql	`65.78% <ø> (+0.14%)`	⬆️
appsec-integration	`51.94% <0.00%> (+6.35%)`	⬆️
appsec-kafka_ldapjs_lodash	`59.17% <ø> (+0.03%)`	⬆️
appsec-mongodb-core_mongoose_mysql	`62.41% <ø> (+0.12%)`	⬆️
appsec-next	`52.88% <ø> (-0.01%)`	⬇️
appsec-node-serialize_passport_postgres	`62.04% <ø> (+0.12%)`	⬆️
appsec-sourcing_stripe_template	`60.43% <ø> (+0.05%)`	⬆️
debugger	`65.70% <ø> (-0.06%)`	⬇️
instrumentations-bucket-0	`48.88% <ø> (+0.08%)`	⬆️
instrumentations-bucket-1	`55.08% <ø> (+0.07%)`	⬆️
instrumentations-bucket-10	`56.71% <ø> (+0.04%)`	⬆️
instrumentations-bucket-11	`48.87% <ø> (+0.08%)`	⬆️
instrumentations-bucket-12	`49.05% <ø> (+0.15%)`	⬆️
instrumentations-bucket-13	`48.81% <ø> (+0.08%)`	⬆️
instrumentations-bucket-2	`50.02% <ø> (+0.23%)`	⬆️
instrumentations-bucket-3	`53.99% <ø> (+0.11%)`	⬆️
instrumentations-bucket-4	`49.29% <ø> (+0.08%)`	⬆️
instrumentations-bucket-5	`53.16% <ø> (+0.10%)`	⬆️
instrumentations-bucket-6	`55.73% <ø> (+0.07%)`	⬆️
instrumentations-bucket-7	`53.63% <ø> (-0.03%)`	⬇️
instrumentations-bucket-8	`54.76% <ø> (+0.07%)`	⬆️
instrumentations-bucket-9	`56.19% <ø> (+0.04%)`	⬆️
instrumentations-instrumentation-couchbase	`48.27% <ø> (+0.04%)`	⬆️
instrumentations-integration-esbuild	`34.06% <0.00%> (+0.09%)`	⬆️
llmobs-ai_anthropic_bedrock	`57.77% <ø> (+<0.01%)`	⬆️
llmobs-bucket-1	`57.11% <ø> (+0.28%)`	⬆️
llmobs-openai	`57.50% <ø> (-0.10%)`	⬇️
llmobs-sdk	`60.07% <ø> (+0.07%)`	⬆️
llmobs-vertex-ai	`54.31% <ø> (+0.06%)`	⬆️
master-coverage	`96.65% <100.00%> (?)`
openfeature	`54.70% <ø> (+0.07%)`	⬆️
openfeature-unit	`49.91% <ø> (+0.03%)`	⬆️
platform-core_esbuild_instrumentations-misc	`38.54% <0.00%> (+0.05%)`	⬆️
platform-integration	`62.33% <ø> (+0.09%)`	⬆️
platform-shimmer_unit-guardrails_webpack	`37.33% <0.00%> (+0.02%)`	⬆️
plugins-bucket-0	`53.26% <ø> (+0.03%)`	⬆️
plugins-bucket-1	`55.20% <ø> (+0.10%)`	⬆️
plugins-bucket-11	`57.53% <ø> (+1.89%)`	⬆️
plugins-bucket-18	`57.14% <ø> (+0.02%)`	⬆️
plugins-bucket-19	`55.44% <ø> (+0.04%)`	⬆️
plugins-bucket-20	`57.66% <ø> (+0.17%)`	⬆️
plugins-bucket-4	`53.87% <ø> (-0.03%)`	⬇️
plugins-bullmq_cassandra_cookie	`57.47% <ø> (+0.03%)`	⬆️
plugins-cookie-parser_crypto_dd-trace-api	`52.42% <ø> (+0.05%)`	⬆️
plugins-fetch_fs_generic-pool	`54.59% <ø> (+<0.01%)`	⬆️
plugins-google-cloud-pubsub_grpc_handlebars	`59.92% <ø> (+0.01%)`	⬆️
plugins-hapi_hono_ioredis	`55.86% <ø> (-0.05%)`	⬇️
plugins-jest_knex_langgraph	`51.77% <ø> (+0.05%)`	⬆️
plugins-ldapjs_light-my-request_limitd-client	`53.54% <ø> (+0.07%)`	⬆️
plugins-lodash_mariadb_memcached	`54.30% <ø> (+0.04%)`	⬆️
plugins-moleculer_mongodb_mongodb-core	`57.40% <ø> (+0.07%)`	⬆️
plugins-mongoose_multer_mysql	`54.79% <ø> (+0.03%)`	⬆️
plugins-mysql2_nats_node-serialize	`56.33% <ø> (+0.07%)`	⬆️
plugins-opensearch_passport-http_pino	`55.19% <ø> (+0.09%)`	⬆️
plugins-postgres_process_pug	`54.38% <ø> (+0.04%)`	⬆️
plugins-redis_router_sequelize	`57.19% <ø> (+0.14%)`	⬆️
plugins-test-and-upstream-rhea_undici_url	`57.06% <ø> (-0.01%)`	⬇️
plugins-valkey_vm_winston	`54.02% <ø> (+0.13%)`	⬆️
plugins-ws	`54.93% <ø> (+0.08%)`	⬆️
profiling	`58.26% <ø> (+0.02%)`	⬆️
serverless-aws-sdk-aws-sdk	`50.78% <ø> (+0.07%)`	⬆️
serverless-aws-sdk-bedrockruntime	`50.75% <ø> (+0.02%)`	⬆️
serverless-aws-sdk-client	`52.09% <ø> (+0.07%)`	⬆️
serverless-aws-sdk-dynamodb	`51.80% <ø> (-0.02%)`	⬇️
serverless-aws-sdk-eventbridge	`46.46% <ø> (+0.16%)`	⬆️
serverless-aws-sdk-kinesis	`54.79% <ø> (+0.03%)`	⬆️
serverless-aws-sdk-lambda	`52.87% <ø> (+0.04%)`	⬆️
serverless-aws-sdk-s3	`51.59% <ø> (+0.01%)`	⬆️
serverless-aws-sdk-serverless-peer-service	`54.82% <ø> (+0.04%)`	⬆️
serverless-aws-sdk-sns	`55.54% <ø> (+0.01%)`	⬆️
serverless-aws-sdk-sqs	`55.96% <ø> (+<0.01%)`	⬆️
serverless-aws-sdk-stepfunctions	`51.36% <ø> (+0.06%)`	⬆️
serverless-aws-sdk-util	`48.54% <ø> (+0.01%)`	⬆️
serverless-bucket-0	`55.24% <ø> (+<0.01%)`	⬆️
serverless-bucket-1	`56.06% <ø> (+0.01%)`	⬆️
test-optimization-cucumber	`72.95% <100.00%> (-0.03%)`	⬇️
test-optimization-cypress	`66.37% <ø> (+0.05%)`	⬆️
test-optimization-jest	`74.29% <ø> (+0.35%)`	⬆️
test-optimization-mocha	`74.69% <ø> (+0.43%)`	⬆️
test-optimization-playwright-playwright-atr	`61.51% <ø> (-0.08%)`	⬇️
test-optimization-playwright-playwright-efd	`61.69% <ø> (-0.08%)`	⬇️
test-optimization-playwright-playwright-final-status	`61.67% <ø> (+0.01%)`	⬆️
test-optimization-playwright-playwright-impacted-tests	`61.61% <ø> (+0.32%)`	⬆️
test-optimization-playwright-playwright-reporting	`61.28% <ø> (-0.08%)`	⬇️
test-optimization-playwright-playwright-test-management	`62.23% <ø> (-0.14%)`	⬇️
test-optimization-playwright-playwright-test-span	`61.42% <ø> (-0.13%)`	⬇️
test-optimization-selenium	`60.72% <100.00%> (-0.07%)`	⬇️
test-optimization-testopt	`59.16% <100.00%> (+0.09%)`	⬆️
test-optimization-vitest	`71.42% <ø> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

fix(test-optimization): support cucumber 13.1 results

781b5d5

github-actions Bot added semver-patch test-optimization labels Jul 15, 2026

juan-fernandez marked this pull request as ready for review July 15, 2026 08:59

juan-fernandez requested review from a team as code owners July 15, 2026 08:59

juan-fernandez requested review from tlhunter and removed request for a team July 15, 2026 08:59

datadog-official Bot reviewed Jul 15, 2026

View reviewed changes

juan-fernandez enabled auto-merge (squash) July 15, 2026 12:45

gnufede approved these changes Jul 15, 2026

View reviewed changes

juan-fernandez merged commit 57429c3 into master Jul 15, 2026
749 of 751 checks passed

juan-fernandez deleted the juan-fernandez/fix-cucumber-13-1-result branch July 15, 2026 13:00

dd-octo-sts Bot pushed a commit that referenced this pull request Jul 16, 2026

fix(test-optimization): support cucumber 13.1 results (#9370)

9518b5a

dd-octo-sts Bot mentioned this pull request Jul 16, 2026

v6.4.0 proposal #9368

Merged

dd-octo-sts Bot pushed a commit that referenced this pull request Jul 16, 2026

fix(test-optimization): support cucumber 13.1 results (#9370)

5be63e0

dd-octo-sts Bot mentioned this pull request Jul 16, 2026

v5.115.0 proposal #9369

Merged

juan-fernandez added a commit that referenced this pull request Jul 16, 2026

fix(test-optimization): support cucumber 13.1 results (#9370)

88cbc98

juan-fernandez added a commit that referenced this pull request Jul 16, 2026

fix(test-optimization): support cucumber 13.1 results (#9370)

1cd8850

Uh oh!

Conversation

juan-fernandez commented Jul 15, 2026

What does this PR do?

Motivation

Additional Notes

Uh oh!

dd-octo-sts Bot commented Jul 15, 2026

Overall package size

Uh oh!

datadog-official Bot commented Jul 15, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Jul 15, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Jul 15, 2026

Benchmarks

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:appsec-appsec-enabled-24

scenario:appsec-appsec-enabled-26

scenario:appsec-appsec-enabled-with-attacks-24

scenario:appsec-appsec-enabled-with-attacks-26

scenario:appsec-control-20

scenario:appsec-control-24

scenario:appsec-control-26

scenario:appsec-iast-no-vulnerability-control-20

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

scenario:appsec-iast-with-vulnerability-control-20

scenario:debugger-line-probe-with-snapshot-default-24

scenario:debugger-line-probe-with-snapshot-minimal-24

scenario:debugger-line-probe-with-snapshot-minimal-26

scenario:debugger-line-probe-without-snapshot-26

scenario:dogstatsd-with-tags-20

scenario:plugin-claude-agent-sdk-compact-stream-scan-20

scenario:plugin-graphql-long-with-depth-on-max-20

scenario:plugin-memcached-get-26

scenario:test-optimization-large-suite-20

Uh oh!

codecov Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-official Bot commented Jul 15, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

datadog-official Bot commented Jul 15, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

codecov Bot commented Jul 15, 2026 •

edited

Loading