[APPSEC-62616] Add inferred proxy spans by Strech · Pull Request #5681 · DataDog/dd-trace-rb

Strech · 2026-05-05T13:23:51Z

What does this PR do?

It's adding inferred spans for apps behind the AWS Gateway.

Motivation:

This is missing part for the tracer that comes to from Endpoint Discovery & Correlation from Inferred Spans RFC.

Change log entry

Yes. Add inferred proxy spans for AWS Gateway.

Additional Notes:

I've enforced ST, to be on the safe side
We don't have a way now to propagate arbitrary tags and I have to put some AppSec tags into propagation, but it's just names, maybe we should come up with communication between products via standard interface (like active support notification, it's on my list)

How to test the change?

CI + ST

datadog-prod-us1-5 · 2026-05-05T13:47:09Z

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 97.15% (+0.02%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0fb0d0d | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-05-05T14:56:02Z

Benchmarks

Benchmark execution time: 2026-05-12 16:04:25

Comparing candidate commit 0fb0d0d in PR branch appsec-62616-inferred-proxy-span with baseline commit 421d0e9 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 98704bb977

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Add tests for: unknown proxy type, absent request timing, partial optional headers, user_agent propagation, quoted region (API Gateway v1), and missing HTTP method fallback. Fix double invocation in apigateway context and strengthen priority test assertions. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

- Replace AppSec::Ext constant with string literals to avoid module coupling between tracing and appsec layers - Strip single quotes from region header (API Gateway v1 format) - Guard resource construction when HTTP method header is absent - Only pass start_time to trace when present (avoid nil in options) - Restore original comments in request_queuing code path - Mark new proxy constants as @!visibility private Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

All other tracers (Go, .NET, PHP, Python, Java) propagate exceptions to the inferred proxy parent span. Add rescue Exception matching the pattern in Rack::TraceMiddleware so the proxy span carries error tags. Force inferred proxy system-tests in CI via forced-tests-list.cfg. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Respects user-configured http_error_statuses by copying the rack span's status directly rather than re-evaluating the status code. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…1970 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Skip inferred span entirely when x-dd-proxy-request-time-ms is missing or non-numeric instead of defaulting to current time. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Only set trace resource to proxy resource when no framework (Rails, Sinatra) has already claimed it, using resource_override? guard matching Rack::TraceMiddleware behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…riable Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Strech · 2026-05-11T14:49:53Z

System tests require update to the latest SHA. ~~Will wait for tomorrow~~

I did an update via workflow

zarirhamza

From a functionality POV, the PR creates the desired experience for APIGW. Serverless approves!

Co-authored-by: Victor Pellan <[email protected]>

dd-octo-sts Bot added integrations Involves tracing integrations tracing labels May 5, 2026

Strech force-pushed the appsec-62616-inferred-proxy-span branch 2 times, most recently from cbf4f0e to 98704bb Compare May 8, 2026 18:34

Strech marked this pull request as ready for review May 8, 2026 18:41

Strech requested review from a team as code owners May 8, 2026 18:41

Strech requested review from mabdinur and y9v May 8, 2026 18:41

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

Comment thread lib/datadog/tracing/contrib/rack/trace_proxy_middleware.rb Outdated

Comment thread lib/datadog/tracing/contrib/rack/trace_proxy_middleware.rb Outdated

wantsui reviewed May 8, 2026

View reviewed changes

Comment thread lib/datadog/tracing/contrib/rack/trace_proxy_middleware.rb Outdated

dd-octo-sts Bot added the core Involves Datadog core libraries label May 11, 2026

y9v approved these changes May 11, 2026

View reviewed changes

Comment thread lib/datadog/tracing/contrib/rack/trace_proxy_middleware.rb Outdated

Strech and others added 14 commits May 11, 2026 15:57

Add inferred span in Tracing Rack contrib

9aa3cdd

Add RBS types for Rack inferred proxy span

65a7cf9

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Refactor proxy middleware contrib

e00e8a4

Use Rack::env type alias in TraceProxyMiddleware RBS

ad7aae0

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Guard against empty x-dd-proxy header bypassing request_queuing

0cb4ea5

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Mirror rack span error status instead of hardcoding >= 500

ab79be0

Respects user-configured http_error_statuses by copying the rack span's status directly rather than re-evaluating the status code. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Skip malformed x-dd-proxy-request-time-ms instead of producing epoch …

be72517

…1970 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Require valid timestamp for inferred proxy span, matching Go/Java

33234e7

Skip inferred span entirely when x-dd-proxy-request-time-ms is missing or non-numeric instead of defaulting to current time. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Rename method to http_method to avoid shadowing Kernel#method

5e3ed1d

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Add note about ARN prefix assumption for non-AWS proxy types

608df7e

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Strech and others added 7 commits May 11, 2026 15:57

Add test for absent x-dd-proxy-domain-name header

f56f643

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Use Rack::response return type in TraceProxyMiddleware RBS

b1e4728

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Refactor call_with_inferred_proxy: early returns first, drop extra va…

2604920

…riable Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Add DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED

b94b49c

Add inferred proxy config condition to the rack

40a8f60

Align trace resource with other libs

e848d3c

Apply code review suggestions

e11ef3b

Strech force-pushed the appsec-62616-inferred-proxy-span branch from adce2c3 to e11ef3b Compare May 11, 2026 13:59

Strech mentioned this pull request May 11, 2026

[🤖] Update System Tests #5734

Merged

Strech added 2 commits May 11, 2026 17:43

Merge branch 'master' into appsec-62616-inferred-proxy-span

a444f09

Merge branch 'master' into appsec-62616-inferred-proxy-span

b6c483c

zarirhamza reviewed May 12, 2026

View reviewed changes

zarirhamza approved these changes May 12, 2026

View reviewed changes

vpellan approved these changes May 12, 2026

View reviewed changes

Comment thread lib/datadog/tracing/contrib/rack/trace_proxy_middleware.rb Outdated

Strech and others added 3 commits May 12, 2026 17:28

Update lib/datadog/tracing/contrib/rack/trace_proxy_middleware.rb

df77d4e

Co-authored-by: Victor Pellan <[email protected]>

Merge branch 'master' into appsec-62616-inferred-proxy-span

cc8877e

Fix formatting

0fb0d0d

Strech merged commit 95a7a3d into master May 12, 2026
584 checks passed

Strech deleted the appsec-62616-inferred-proxy-span branch May 12, 2026 16:30

dd-octo-sts Bot added this to the 2.33.0 milestone May 12, 2026

Strech mentioned this pull request May 13, 2026

Bump to version 2.33.0 #5746

Merged

sameerank mentioned this pull request May 13, 2026

Add style checks from PR feedback patterns #5764

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[APPSEC-62616] Add inferred proxy spans#5681

[APPSEC-62616] Add inferred proxy spans#5681
Strech merged 26 commits into
masterfrom
appsec-62616-inferred-proxy-span

Strech commented May 5, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-5 Bot commented May 5, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

pr-commenter Bot commented May 5, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Strech commented May 11, 2026 •

edited

Loading

Uh oh!

zarirhamza left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Strech commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-5 Bot commented May 5, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Strech commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zarirhamza left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Strech commented May 5, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented May 5, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

pr-commenter Bot commented May 5, 2026 •

edited

Loading

Strech commented May 11, 2026 •

edited

Loading