Skip to content

APMSP-3553 DI: redactor normalize Rack CGI header prefix#5911

Merged
p-datadog merged 4 commits into
masterfrom
apmsp-3553-redactor-rack-header-prefix
Jun 26, 2026
Merged

APMSP-3553 DI: redactor normalize Rack CGI header prefix#5911
p-datadog merged 4 commits into
masterfrom
apmsp-3553-redactor-rack-header-prefix

Conversation

@p-datadog

Copy link
Copy Markdown
Member

What does this PR do?

Makes Redactor#normalize strip a leading http_ so that Rack CGI
header keys (HTTP_AUTHORIZATION, HTTP_COOKIE, HTTP_X_API_KEY,
…) collapse to the same identifier as the bare header name and are
matched against DEFAULT_REDACTED_IDENTIFIERS (authorization,
cookie, xapikey, …).

Motivation

Rack rewrites incoming HTTP header names to the CGI form HTTP_<HEADER>
(dashes to underscores, HTTP_ prefix). The previous normalize
implementation stripped only [-_$@], so Rack-shaped keys collapsed to
httpauthorization, httpcookie, httpxapikey, etc., and no entry in
the default identifier list matched them. Any hash capture that
included a Rack env (every Rack-based middleware, Rails controller,
or Sinatra handler keeps it in lexical scope) ended up emitting those
values verbatim into the snapshot, while the same headers under their
bare names (Authorization, Cookie, X-API-Key) were redacted.
Snapshots should not distinguish between env["HTTP_AUTHORIZATION"]
and a local authorization — both name the same thing.

The change is in normalize rather than DEFAULT_REDACTED_IDENTIFIERS
because the list is shared across tracers and the gap is structural:
every entry in the list has an HTTP_-prefixed twin under Rack, and
maintaining the twin in lockstep is more error-prone than collapsing
the prefix at normalization time. The strip runs after downcase and
is anchored to \Ahttp_, so identifiers that simply start with the
letters http without a following underscore (e.g. httpinfo) are
unaffected. User-supplied identifiers go through the same path, so a
user-defined http_custom entry now also matches the bare custom
form — the same collapsing already applies to the punctuation gsub
(pass_word matches password), and this extends the principle to
the Rack prefix.

Change log entry

Yes. Dynamic Instrumentation: redactor normalizes the Rack CGI header
prefix (HTTP_<HEADER>), so snapshot captures of a Rack environment
redact sensitive header values under the same defaults as their bare
header names.

Implementation

Redactor#normalize becomes:

str.to_s.strip.downcase.sub(/\Ahttp_/, "").gsub(/[-_$@]/, "")

The sub runs once at normalization time; there is no per-lookup
cost beyond a single anchored regex check on already-downcased input.
Both the default list and the user-defined / excluded lists are
normalized with the same function (see the redacted_identifiers
private method), so the prefix strip applies symmetrically — a name
excluded via redaction_excluded_identifiers continues to suppress
redaction regardless of whether the captured key is the bare form or
the HTTP_-prefixed form.

How to test the change?

bundle exec rspec spec/datadog/di/redactor_spec.rb

49 examples, 0 failures.

bundle exec rake test:di:di_with_ext

808 examples, 0 failures, 8 pending (same 8 Ruby-version-gated pending
tests as on master).

New spec cases live under describe '#redact_identifier?' and cover:

  • HTTP_AUTHORIZATION, HTTP_COOKIE, HTTP_X_API_KEY,
    HTTP_X_AUTH_TOKEN, HTTP_SET_COOKIE — all match their bare
    counterparts in DEFAULT_REDACTED_IDENTIFIERS
  • HTTP_REFERER, HTTP_HOST — Rack-shaped but not in the list;
    pass through unredacted as controls
  • httpinfo — boundary case proving the prefix strip is anchored to
    the literal CGI form (leading http_), not arbitrary identifiers
    that begin with the letters http
  • HTTP_CUSTOM and custom against a user-defined http_custom
    entry — confirms the user-defined list normalizes through the same
    function

`Redactor#normalize` previously stripped only `[-_$@]` before matching
against `DEFAULT_REDACTED_IDENTIFIERS`. Hash keys originating from a
Rack environment use the CGI form `HTTP_<HEADER>` (Rack rewrites
incoming HTTP headers, converting dashes to underscores and prepending
`HTTP_`). Those keys collapsed to `httpauthorization`, `httpcookie`,
`httpxapikey`, etc. — none of which appear in the default list — so
the corresponding values were not matched against entries like
`authorization`, `cookie`, or `xapikey`. Any snapshot or line probe
placed where a Rack `env` hash is in lexical scope (every Rack-based
middleware, Rails controller, or Sinatra handler) emitted those
values unredacted.

Strip a leading `http_` in `normalize` before the punctuation gsub.
The sub runs after `downcase`, so it only matches the literal Rack
CGI form — identifiers that happen to start with the letters `http`
without a following underscore (e.g. `httpinfo`) are unaffected. This
generalizes over every entry in the default list, so future additions
do not need a parallel `HTTP_`-prefixed twin maintained in lockstep.

Specs added to `spec/datadog/di/redactor_spec.rb` cover the canonical
Rack header forms (`HTTP_AUTHORIZATION`, `HTTP_COOKIE`,
`HTTP_X_API_KEY`, `HTTP_X_AUTH_TOKEN`, `HTTP_SET_COOKIE`), a
non-redacted control (`HTTP_REFERER`, `HTTP_HOST`), the `http`-without-
underscore boundary case, and the user-defined-list interaction (a
user-supplied identifier prefixed with `http_` matches both the CGI
form and the bare form).

Verified:
- `bundle exec rspec spec/datadog/di/redactor_spec.rb`
  49 examples, 0 failures
- `bundle exec rake test:di:di_with_ext`
  808 examples, 0 failures, 8 pending (pre-existing)
- `bundle exec rake typecheck`
  No type error detected
- `bundle exec rake standard`
  No offenses
@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jun 16, 2026
@dd-octo-sts dd-octo-sts Bot added the debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database) label Jun 16, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 16, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 90.00% (+0.13%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5511be4 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 16, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-26 19:37:52

Comparing candidate commit 5511be4 in PR branch apmsp-3553-redactor-rack-header-prefix with baseline commit 8f40a35 in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:profiling - Allocations ()

  • 🟩 throughput [+161371.191op/s; +200633.522op/s] or [+5.327%; +6.623%]

p-ddsign added 2 commits June 18, 2026 12:01
The previous commit makes Redactor#normalize strip a leading 'http_'
so Rack CGI-form keys (HTTP_AUTHORIZATION, ...) match the default
redaction list under their bare names. Because the same normalize
runs over redaction_excluded_identifiers, excluding a bare name
symmetrically suppresses redaction of its HTTP_-prefixed Rack form
— but the prior spec only covered the bare-form exclusion.

Add a spec case in the 'when excluded identifiers are specified'
context that excludes 'password' and asserts HTTP_PASSWORD is also
not redacted, locking down the symmetry the PR description claims.

bundle exec rspec spec/datadog/di/redactor_spec.rb
  50 examples, 0 failures
The comment attributed the httpinfo exclusion to running the strip before
the punctuation gsub. That protection actually comes from the \Ahttp_
regex requiring an underscore after http. The real reason for the
ordering is that the gsub removes the underscore, so a later \Ahttp_
match would never fire. Reword the comment to state both facts
accurately.
@p-datadog
p-datadog marked this pull request as ready for review June 25, 2026 00:17
@p-datadog
p-datadog requested a review from a team as a code owner June 25, 2026 00:17
@p-datadog
p-datadog requested a review from Copilot June 25, 2026 00:17
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Dynamic Instrumentation (DI) snapshot redaction by updating Redactor#normalize to treat Rack CGI-style header keys (HTTP_<HEADER>) as equivalent to their bare header names, ensuring default sensitive identifiers (e.g., authorization, cookie, xapikey) correctly redact values captured from a Rack env.

Changes:

  • Update Redactor#normalize to strip a leading http_ (after downcasing) before applying the existing punctuation removal.
  • Add RSpec coverage for Rack CGI header keys redacting the same as their bare counterparts, including boundary/control cases and user-defined identifier behavior.
  • Add coverage that the same normalization applies to the exclusion list (e.g., excluding password also excludes HTTP_PASSWORD).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
lib/datadog/di/redactor.rb Adjusts identifier normalization to collapse Rack CGI HTTP_ header keys to the same canonical form used by default/user redaction lists.
spec/datadog/di/redactor_spec.rb Adds targeted tests validating redaction/exclusion behavior for HTTP_* keys and boundary cases.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 471af66264

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/datadog/di/redactor.rb Outdated
@p-datadog
p-datadog merged commit 4d54496 into master Jun 26, 2026
591 checks passed
@p-datadog
p-datadog deleted the apmsp-3553-redactor-rack-header-prefix branch June 26, 2026 20:28
@dd-octo-sts dd-octo-sts Bot added this to the 2.37.0 milestone Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos debugger Live Debugger (+Dynamic Instrumentation, +Symbol Database)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants