Skip to content

fix(test-optimization): preserve zero EFD retry budgets#9341

Merged
juan-fernandez merged 1 commit into
masterfrom
BridgeAR/2026-07-14-preserve-zero-efd-retries
Jul 15, 2026
Merged

fix(test-optimization): preserve zero EFD retry budgets#9341
juan-fernandez merged 1 commit into
masterfrom
BridgeAR/2026-07-14-preserve-zero-efd-retries

Conversation

@BridgeAR

Copy link
Copy Markdown
Member

Summary

Falsy fallback logic replaced explicit zero EFD retry budgets with defaults, scheduling retries the backend disabled.

Test plan

  • Run settings-parser and retry-budget unit specs
  • Run the Cucumber parallel regression through agentless and EVP proxy modes
  • Check changed-line and branch coverage

Falsy fallback logic replaced an explicit zero retry budget with defaults, scheduling retries the backend disabled.
@dd-octo-sts

dd-octo-sts Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.77 MB
Deduped: 7.43 MB
No deduping: 7.43 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@BridgeAR
BridgeAR marked this pull request as ready for review July 14, 2026 16:41
@BridgeAR
BridgeAR requested a review from a team as a code owner July 14, 2026 16:41
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 14, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 2 tests - 2 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 96.69% (+0.08%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fbadc6d | Docs | Datadog PR Page | Give us feedback!

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

All-zero EFD retry budgets are now correctly preserved instead of being replaced with defaults, preventing unwanted test retries when the backend explicitly disables retries. The fix uses nullish coalescing (??) to distinguish zero from undefined, and getMaxEfdRetryCount() now returns undefined for empty configurations. All six retry-count tests pass, including new tests that lock in the correct zero-preservation behavior.

Was this helpful? React 👍 or 👎

📊 Validated against 10 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit fbadc6d · What is Autotest? · Any feedback? Reach out in #autotest

@pr-commenter

pr-commenter Bot commented Jul 14, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-14 16:49:12

Comparing candidate commit fbadc6d in PR branch BridgeAR/2026-07-14-preserve-zero-efd-retries with baseline commit 0ad9953 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2320 metrics, 38 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-209.167ms; +203.846ms] or [-7.880%; +7.679%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-234.456ms; +232.210ms] or [-9.162%; +9.074%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-161.123ms; +175.760ms] or [-5.218%; +5.692%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-186884.370µs; +187998.303µs] or [-6.439%; +6.477%]

scenario:appsec-control-20

  • unstable execution_time [-126.603ms; +130.567ms] or [-7.656%; +7.896%]

scenario:appsec-control-24

  • unstable execution_time [-114586.324µs; +114775.224µs] or [-9.232%; +9.248%]

scenario:appsec-control-26

  • unstable execution_time [-134.854ms; +114.619ms] or [-10.955%; +9.311%]

scenario:appsec-iast-no-vulnerability-control-20

  • unstable execution_time [-10.111ms; +18.504ms] or [-3.951%; +7.230%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-18.349ms; +20.482ms] or [-6.998%; +7.811%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-30082.374µs; +28494.159µs] or [-5.430%; +5.144%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1658.449ms; +660.246ms] or [-19.509%; +7.767%]
  • unstable execution_time [-1684.984ms; +637.248ms] or [-18.305%; +6.923%]
  • unstable instructions [-15.9G instructions; +5.8G instructions] or [-22.679%; +8.226%]
  • unstable throughput [-197.436op/s; +459.027op/s] or [-5.530%; +12.857%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-3699.505ms; +5279.341ms] or [-29.232%; +41.715%]
  • unstable execution_time [-3778.736ms; +5390.529ms] or [-28.213%; +40.246%]
  • unstable instructions [-32.8G instructions; +46.7G instructions] or [-30.680%; +43.616%]
  • unstable max_rss_usage [-12.273MB; +16.993MB] or [-7.270%; +10.066%]
  • unstable throughput [-1046.286op/s; +741.597op/s] or [-39.934%; +28.305%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-2360.695ms; +3436.793ms] or [-25.574%; +37.232%]
  • unstable execution_time [-2331.147ms; +3426.359ms] or [-23.441%; +34.454%]
  • unstable instructions [-20.2G instructions; +28.9G instructions] or [-26.702%; +38.323%]
  • unstable max_rss_usage [-9.323MB; +13.028MB] or [-5.812%; +8.121%]
  • unstable throughput [-966.653op/s; +653.052op/s] or [-28.595%; +19.318%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2587.727ms; +4131.165ms] or [-27.092%; +43.251%]
  • unstable execution_time [-2601.820ms; +4147.114ms] or [-25.359%; +40.420%]
  • unstable instructions [-23.2G instructions; +36.9G instructions] or [-29.143%; +46.414%]
  • unstable max_rss_usage [-7.703MB; +13.221MB] or [-4.835%; +8.299%]
  • unstable throughput [-815.464op/s; +511.311op/s] or [-25.252%; +15.833%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-309.904ms; +385.405ms] or [-6.507%; +8.092%]
  • unstable execution_time [-310.325ms; +381.524ms] or [-6.410%; +7.881%]
  • unstable throughput [-135049.964op/s; +109839.934op/s] or [-7.786%; +6.332%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-702.837ms; +753.128ms] or [-5.475%; +5.867%]
  • unstable execution_time [-712.509ms; +755.503ms] or [-5.429%; +5.757%]
  • unstable throughput [-3.450op/s; +3.280op/s] or [-5.626%; +5.349%]

scenario:plugin-pg-service-26

  • unstable cpu_usage_percentage [-6.592%; +3.956%]
  • unstable execution_time [-50.983ms; +71.426ms] or [-5.706%; +7.994%]
  • unstable throughput [-436698.526op/s; +321474.389op/s] or [-6.422%; +4.728%]

@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.66%. Comparing base (0ad9953) to head (fbadc6d).
⚠️ Report is 19 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9341      +/-   ##
==========================================
+ Coverage   96.61%   96.66%   +0.05%     
==========================================
  Files         920      923       +3     
  Lines      122236   122939     +703     
  Branches    21070    21598     +528     
==========================================
+ Hits       118097   118845     +748     
+ Misses       4139     4094      -45     
Flag Coverage Δ
aiguard 54.20% <25.00%> (+0.60%) ⬆️
aiguard-integration 58.08% <25.00%> (+1.11%) ⬆️
apm-bucket-0 54.44% <25.00%> (+0.61%) ⬆️
apm-bucket-1 59.38% <25.00%> (+1.28%) ⬆️
apm-bucket-2 56.89% <25.00%> (+0.75%) ⬆️
apm-capabilities-tracing 60.82% <100.00%> (+0.44%) ⬆️
apm-integrations-aerospike 53.59% <25.00%> (+0.61%) ⬆️
apm-integrations-confluentinc-kafka-javascript 58.12% <25.00%> (+1.03%) ⬆️
apm-integrations-couchbase 53.92% <25.00%> (+0.61%) ⬆️
apm-integrations-http 59.03% <25.00%> (+1.05%) ⬆️
apm-integrations-kafkajs 58.78% <25.00%> (+1.03%) ⬆️
apm-integrations-next 55.01% <25.00%> (+0.73%) ⬆️
apm-integrations-prisma 54.78% <25.00%> (+0.57%) ⬆️
apm-integrations-tedious 53.70% <25.00%> (+0.62%) ⬆️
appsec 69.76% <25.00%> (+0.80%) ⬆️
appsec-express_fastify_graphql 66.89% <25.00%> (+1.18%) ⬆️
appsec-integration 52.77% <22.22%> (+0.91%) ⬆️
appsec-kafka_ldapjs_lodash 60.04% <25.00%> (+0.93%) ⬆️
appsec-mongodb-core_mongoose_mysql 63.32% <25.00%> (+1.06%) ⬆️
appsec-next 53.55% <25.00%> (+0.66%) ⬆️
appsec-node-serialize_passport_postgres 63.00% <25.00%> (+1.11%) ⬆️
appsec-sourcing_stripe_template 61.40% <25.00%> (+1.05%) ⬆️
debugger 66.37% <25.00%> (+0.59%) ⬆️
instrumentations-bucket-0 49.44% <25.00%> (+0.64%) ⬆️
instrumentations-bucket-1 56.06% <25.00%> (+1.06%) ⬆️
instrumentations-bucket-10 57.71% <25.00%> (+1.06%) ⬆️
instrumentations-bucket-11 49.44% <25.00%> (+0.65%) ⬆️
instrumentations-bucket-12 49.67% <25.00%> (+0.78%) ⬆️
instrumentations-bucket-13 49.38% <25.00%> (+0.65%) ⬆️
instrumentations-bucket-2 50.80% <25.00%> (+1.01%) ⬆️
instrumentations-bucket-3 55.04% <25.00%> (+1.17%) ⬆️
instrumentations-bucket-4 49.85% <25.00%> (+0.64%) ⬆️
instrumentations-bucket-5 54.28% <25.00%> (+1.23%) ⬆️
instrumentations-bucket-6 56.76% <25.00%> (+1.12%) ⬆️
instrumentations-bucket-7 54.39% <25.00%> (+0.74%) ⬆️
instrumentations-bucket-8 55.71% <25.00%> (+1.04%) ⬆️
instrumentations-bucket-9 57.21% <25.00%> (+1.08%) ⬆️
instrumentations-instrumentation-couchbase 48.84% <25.00%> (+0.66%) ⬆️
instrumentations-integration-esbuild 34.48% <22.22%> (+0.59%) ⬆️
llmobs-ai_anthropic_bedrock 58.46% <25.00%> (+0.69%) ⬆️
llmobs-bucket-1 57.00% <25.00%> (-0.04%) ⬇️
llmobs-openai 58.10% <25.00%> (+0.67%) ⬆️
llmobs-sdk 61.00% <25.00%> (+0.99%) ⬆️
llmobs-vertex-ai 54.86% <25.00%> (+0.60%) ⬆️
master-coverage 96.66% <100.00%> (?)
openfeature 54.59% <25.00%> (-0.02%) ⬇️
openfeature-unit 50.46% <25.00%> (+0.63%) ⬆️
platform-core_esbuild_instrumentations-misc 38.37% <22.22%> (-0.03%) ⬇️
platform-integration 63.48% <25.00%> (+1.15%) ⬆️
platform-shimmer_unit-guardrails_webpack 37.20% <22.22%> (-0.03%) ⬇️
plugins-bucket-0 54.00% <25.00%> (+0.78%) ⬆️
plugins-bucket-1 56.36% <25.00%> (+1.34%) ⬆️
plugins-bucket-11 58.77% <25.00%> (+1.27%) ⬆️
plugins-bucket-18 58.41% <25.00%> (+1.30%) ⬆️
plugins-bucket-19 55.36% <25.00%> (-0.06%) ⬇️
plugins-bucket-20 58.38% <25.00%> (+0.91%) ⬆️
plugins-bucket-4 53.83% <25.00%> (-0.06%) ⬇️
plugins-bullmq_cassandra_cookie 57.33% <25.00%> (-0.06%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 52.30% <25.00%> (-0.06%) ⬇️
plugins-fetch_fs_generic-pool 54.44% <25.00%> (-0.13%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 59.83% <25.00%> (-0.05%) ⬇️
plugins-hapi_hono_ioredis 55.84% <25.00%> (-0.06%) ⬇️
plugins-jest_knex_langgraph 51.65% <25.00%> (-0.06%) ⬇️
plugins-ldapjs_light-my-request_limitd-client 53.43% <25.00%> (-0.06%) ⬇️
plugins-lodash_mariadb_memcached 54.19% <25.00%> (-0.06%) ⬇️
plugins-moleculer_mongodb_mongodb-core 57.27% <25.00%> (-0.06%) ⬇️
plugins-mongoose_multer_mysql 54.69% <25.00%> (-0.06%) ⬇️
plugins-mysql2_nats_node-serialize 56.18% <25.00%> (-0.06%) ⬇️
plugins-opensearch_passport-http_pino 55.03% <25.00%> (-0.06%) ⬇️
plugins-postgres_process_pug 54.27% <25.00%> (-0.06%) ⬇️
plugins-redis_router_sequelize 56.98% <25.00%> (-0.06%) ⬇️
plugins-test-and-upstream-rhea_undici_url 57.00% <25.00%> (-0.03%) ⬇️
plugins-valkey_vm_winston 53.91% <25.00%> (-0.06%) ⬇️
plugins-ws 54.77% <25.00%> (-0.06%) ⬇️
profiling 58.19% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-aws-sdk 50.66% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-bedrockruntime 50.68% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-client 51.92% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-dynamodb 51.74% <25.00%> (-0.06%) ⬇️
serverless-aws-sdk-eventbridge 46.21% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-kinesis 54.69% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-lambda 52.77% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-s3 51.52% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-serverless-peer-service 54.71% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-sns 55.46% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-sqs 55.89% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-stepfunctions 51.24% <25.00%> (-0.05%) ⬇️
serverless-aws-sdk-util 48.42% <25.00%> (-0.06%) ⬇️
serverless-bucket-0 55.16% <25.00%> (-0.02%) ⬇️
serverless-bucket-1 56.00% <25.00%> (-0.06%) ⬇️
test-optimization-cucumber 72.92% <100.00%> (-0.01%) ⬇️
test-optimization-cypress 66.30% <100.00%> (+0.05%) ⬆️
test-optimization-jest 74.22% <100.00%> (-0.03%) ⬇️
test-optimization-mocha 74.67% <100.00%> (+0.06%) ⬆️
test-optimization-playwright-playwright-atr 61.53% <100.00%> (+0.09%) ⬆️
test-optimization-playwright-playwright-efd 61.72% <100.00%> (+0.03%) ⬆️
test-optimization-playwright-playwright-final-status 61.70% <100.00%> (+0.03%) ⬆️
test-optimization-playwright-playwright-impacted-tests 61.42% <100.00%> (+0.20%) ⬆️
test-optimization-playwright-playwright-reporting 61.30% <100.00%> (+0.03%) ⬆️
test-optimization-playwright-playwright-test-management 62.27% <100.00%> (-0.08%) ⬇️
test-optimization-playwright-playwright-test-span 61.44% <100.00%> (-0.02%) ⬇️
test-optimization-selenium 60.64% <90.00%> (-0.13%) ⬇️
test-optimization-testopt 59.13% <90.00%> (+0.08%) ⬆️
test-optimization-vitest 71.56% <100.00%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@juan-fernandez

Copy link
Copy Markdown
Collaborator

does only cucumber need fixes and an integration test for this bug?

@juan-fernandez juan-fernandez left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sry, I want to run a quick codex review

@juan-fernandez

Copy link
Copy Markdown
Collaborator

@codex review

@juan-fernandez

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: fbadc6da84

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@juan-fernandez
juan-fernandez enabled auto-merge (squash) July 15, 2026 12:37
@juan-fernandez
juan-fernandez merged commit 529ceb2 into master Jul 15, 2026
1279 checks passed
@juan-fernandez
juan-fernandez deleted the BridgeAR/2026-07-14-preserve-zero-efd-retries branch July 15, 2026 12:39
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 16, 2026
Falsy fallback logic replaced an explicit zero retry budget with defaults, scheduling retries the backend disabled.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 16, 2026
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 16, 2026
Falsy fallback logic replaced an explicit zero retry budget with defaults, scheduling retries the backend disabled.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 16, 2026
juan-fernandez pushed a commit that referenced this pull request Jul 16, 2026
Falsy fallback logic replaced an explicit zero retry budget with defaults, scheduling retries the backend disabled.
juan-fernandez pushed a commit that referenced this pull request Jul 16, 2026
Falsy fallback logic replaced an explicit zero retry budget with defaults, scheduling retries the backend disabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants