Skip to content

feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure#4708

Merged
RomainMuller merged 12 commits into
mainfrom
romain.marcadier/APPSEC-62578/asm-rfc1012-consolidation
May 4, 2026
Merged

feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure#4708
RomainMuller merged 12 commits into
mainfrom
romain.marcadier/APPSEC-62578/asm-rfc1012-consolidation

Conversation

@RomainMuller

@RomainMuller RomainMuller commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes the remaining MUST-implement gaps from RFC-1012 — ASM Tags, Metrics & Logs Consolidation in dd-trace-go.

Changes

Slice What
_dd.rc.client_id New span tag emitted once per WAF instance via remoteconfig.ClientID()
waf/rasp.error semantics _dd.appsec.waf.error / rasp.error now emit the closest-to-zero ddwaf_run error code instead of a count; telemetry tag renamed error_code:waf_error: (breaking)
block_failure New _dd.appsec.block.failed=1 span tag, waf.requests{block_failure:bool} telemetry tag, and rasp.rule.match{block:failure} variant when BlockingUnavailable=true
error.type/message _dd.appsec.error.type and _dd.appsec.error.message (≤512 B) set on root span for first WAF/RASP exception per request; all exception log sites now carry RFC-1012 log_type tag + stack trace
log_type vocab local::diagnosticappsec::waf::ruleset::diagnostic; RC subscriber error paths tagged rc::asm::exception
rasp.rule.skipped Metric emitted with reason:after-request when a RASP call arrives after the WAF context is closed
api_security.missing_route + user-auth metrics appsec.api_security.missing_route, instrum.user_auth.missing_user_login, instrum.user_auth.missing_user_id now emitted

Out of scope (per RFC or blocked)

  • _dd.appsec.excluded / request_excluded — blocked on libddwaf
  • sdk.event{signup} — needs user-lifecycle RFC owner sign-off before new SDK surface
  • rasp.rule.duration — not yet registered in DataDog/dd-go common_metrics.json

Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0hEQgyKP54kM1LZU98GL8MaP60tQrA/):
the span tag \`_dd.rc.client_id\` must be set once per WAF instance on the
service entry span so the backend can correlate WAF decisions with the
Remote Configuration client that delivered the rule-set.

- Add \`ClientID() string\` to \`internal/remoteconfig\` — returns the
  singleton client's ID (empty when RC is not started).
- Store \`rcClientID\` on the WAF \`Feature\` struct at construction time
  (captured via \`remoteconfig.ClientID()\`).
- Pass \`rcClientID\` to \`AddRulesMonitoringTags\`, which now emits the
  \`_dd.rc.client_id\` tag when the value is non-empty. The tag is emitted
  exactly once per WAF instance via the existing \`sync.Once\` guard.

JJ-Change-Id: slsvrr
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h
EQgyKP54kM1LZU98GL8MaP60tQrA/): two breaking changes:

1. \`_dd.appsec.waf.error\` and \`_dd.appsec.rasp.error\` span tags were
   emitting a cumulative **count** of errors. The RFC requires the
   closest-to-zero (least-negative) ddwaf_run error code. Replace the
   \`atomic.Uint32\` count fields with \`atomic.Int32\` code accumulators
   (sentinel 0 = no error; updated via CAS to keep the max value, which
   for negative codes is the closest to zero). For RASP the accumulator
   is per rule type; the span tag reports the closest-to-zero across
   all rule types.

2. The telemetry metrics \`appsec.waf.error\` and \`appsec.rasp.error\`
   had a tag \`error_code:<n>\`. The RFC mandates \`waf_error:<n>\`. Hard
   rename applied to both emitters in \`metrics.go\`.

Unit tests added for the CAS accumulator and for the span tag behaviour
(both WAF and RASP paths).

JJ-Change-Id: typulw
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h
EQgyKP54kM1LZU98GL8MaP60tQrA/): when the WAF requests a block but the
framework cannot honour it (\`BlockingUnavailable=true\`), the library must
record the failure rather than falsely claiming a successful block.

Changes:
- \`RequestMilestones\` gains \`blockFailed bool\`; the WAF-scope accumulator
  propagates it so \`waf.requests\` includes a \`block_failure:<bool>\` tag.
- \`ContextOperation\` gets \`SetBlockFailed()\` / \`IsBlockFailed()\`; the
  blocking event handler in \`SetupActionHandlers\` calls \`SetBlockFailed()\`
  instead of setting \`appsec.blocked\` when \`blockingUnavailable=true\`.
- \`run.go\`: milestones now set \`requestBlocked: blocking && !blockFailed\`
  so the two flags are mutually exclusive.
- \`AddWAFMonitoringTags\`: emits \`_dd.appsec.block.failed=1\` when set.
- RASP \`rasp.rule.match\` block tag extended with \`block:failure\` case.
- \`Feature\` stores \`blockingUnavailable\` from \`cfg.BlockingUnavailable\`.
- Existing \`waf.requests\` tag assertions updated to include the new tag.

JJ-Change-Id: nwvzpx
…tags

Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h
EQgyKP54kM1LZU98GL8MaP60tQrA/): when a WAF or RASP run produces a
non-timeout error, record the exception category and a truncated error
message as root span tags so the backend can surface them.

Changes:
- \`ContextMetrics.RecordException(errType, err)\` stores the first error
  per request (first-error-wins via \`sync.Once\`) and emits a telemetry
  log with the RFC-1012 \`log_type\` tag and a stack trace.
- Exception type constants (\`ExceptionTypeWAF\`, \`ExceptionTypeRASP\`,
  \`ExceptionTypeInstrumentation\`) exported from \`emitter/waf\`.
- \`IncWafError\` calls \`RecordException\` with the appropriate type
  (WAF vs RASP vs instrumentation path).
- \`AddWAFMonitoringTags\` emits \`_dd.appsec.error.type\` and
  \`_dd.appsec.error.message\` (≤512 bytes) when an exception was recorded.
- \`NewWAFFeature\`'s libddwaf load warning tagged with
  \`log_type:appsec::waf::exception\` + stack trace.
- Round-tripper body parse warnings tagged with
  \`log_type:appsec::instrumentation::exception\` + stack trace.

JJ-Change-Id: vpwkuq
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h
EQgyKP54kM1LZU98GL8MaP60tQrA/): two log-type renames and two new
exception-typed log sites.

- \`wafmanager.go\`: rename ad-hoc \`log_type:local::diagnostic\` to the
  RFC-mandated \`appsec::waf::ruleset::diagnostic\` on the local-rules
  diagnostic path.
- \`internal/appsec/remoteconfig.go\`: the two existing bare
  \`telemetrylog.Error\` calls in RC subscriber error paths now carry
  \`log_type:rc::asm::exception\` and \`WithStacktrace()\` as required by
  RFC-1012 for exception-typed telemetry logs.

No import cycle: \`internal/remoteconfig\` does not depend on
\`internal/telemetry/log\`, verified via \`go list -deps\`.

JJ-Change-Id: lumlzm
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h
EQgyKP54kM1LZU98GL8MaP60tQrA/): three declared-but-never-emitted metrics
are now populated.

\`appsec.api_security.missing_route\`:
- Emitted in \`httpsec.Feature.OnResponse\` when API security is enabled
  but \`op.Route()\` is empty (framework did not set the matched route).
- Tagged with \`framework:<name>\`.

\`appsec.instrum.user_auth.missing_user_login\`:
- Emitted from \`usersec.Feature.OnFinish\` when the login field is empty
  for \`login_success\` or \`login_failure\` events.

\`appsec.instrum.user_auth.missing_user_id\`:
- Emitted from \`usersec.Feature.OnFinish\` when the user-ID field is
  empty for \`login_success\` or \`authenticated_request\` events.

Both user-auth metrics carry \`event_type:<value>\` and \`framework:<name>\`
tags. The framework is resolved by walking the operation parent chain to
find the nearest \`*httpsec.HandlerOperation\`; falls back to "unknown".

JJ-Change-Id: ryrtoo
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h
EQgyKP54kM1LZU98GL8MaP60tQrA/): the previously declared-but-never-emitted
\`rasp.rule.skipped\` count metric is now populated.

\`rasp.rule.skipped\` (Count):
- A lazily-filled map keyed by (rule_type, reason) is added to
  \`HandleMetrics\`.
- \`ContextMetrics.SkipRASPRule(ruleType, reason)\` is the public entry
  point; it emits one count per skip event.
- \`ContextOperation.Run\` emits \`reason:after-request\` when the WAF
  context has already been closed at the time of a RASP call.

Note: \`rasp.rule.duration\` is not implemented here — it is absent from
DataDog/dd-go's common_metrics.json and therefore not yet in the telemetry
spec. It will be added in a follow-up once the spec is updated.

JJ-Change-Id: tmqsym
Merges all RFC-1012 sibling PRs to verify zero integration conflicts
and a clean combined test run. Not intended for submission.

Parents:
  romain.marcadier/asm-rc-client-id
  romain.marcadier/asm-waf-rasp-error-semantics
  romain.marcadier/asm-block-failure
  romain.marcadier/asm-error-type-message
  romain.marcadier/asm-log-type-vocab
  romain.marcadier/asm-rasp-rule-duration-skipped
  romain.marcadier/asm-route-and-user-auth-metrics

Conflict resolution: hot-spot files touched by multiple PRs
(metrics.go, tags.go, tags_test.go, waf.go) were merged by hand,
taking all additions from every sibling branch.

JJ-Change-Id: kwvkxp
@RomainMuller
RomainMuller requested review from a team as code owners April 30, 2026 09:07
@codecov

codecov Bot commented Apr 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 15.83333% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.07%. Comparing base (997e8b2) to head (3b54935).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/appsec/emitter/waf/metrics.go 14.89% 40 Missing ⚠️
internal/appsec/listener/usersec/usec.go 0.00% 24 Missing ⚠️
internal/appsec/listener/httpsec/roundtripper.go 0.00% 8 Missing ⚠️
internal/appsec/remoteconfig.go 0.00% 8 Missing ⚠️
internal/appsec/listener/waf/waf.go 0.00% 6 Missing ⚠️
internal/remoteconfig/remoteconfig.go 0.00% 6 Missing ⚠️
internal/appsec/emitter/waf/run.go 0.00% 3 Missing ⚠️
internal/appsec/listener/waf/tags.go 80.00% 2 Missing and 1 partial ⚠️
internal/appsec/listener/httpsec/http.go 0.00% 1 Missing and 1 partial ⚠️
internal/appsec/config/wafmanager.go 0.00% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
internal/appsec/config/wafmanager.go 0.00% <0.00%> (ø)
internal/appsec/listener/httpsec/http.go 74.39% <0.00%> (ø)
internal/appsec/emitter/waf/run.go 0.00% <0.00%> (ø)
internal/appsec/listener/waf/tags.go 70.45% <80.00%> (ø)
internal/appsec/listener/waf/waf.go 0.00% <0.00%> (ø)
internal/remoteconfig/remoteconfig.go 63.88% <0.00%> (ø)
internal/appsec/listener/httpsec/roundtripper.go 0.00% <0.00%> (ø)
internal/appsec/remoteconfig.go 54.88% <0.00%> (ø)
internal/appsec/listener/usersec/usec.go 0.00% <0.00%> (ø)
internal/appsec/emitter/waf/metrics.go 3.31% <14.89%> (ø)

... and 441 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Apr 30, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 15.74%
Overall Coverage: 61.37% (-0.16%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3b54935 | Docs | Datadog PR Page | Give us feedback!

Comment thread internal/appsec/emitter/waf/metrics.go Fixed
@eliottness

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e004320f44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/appsec/emitter/waf/run.go Outdated
wafTimeout := errors.Is(err, waferrors.ErrTimeout)
rateLimited := op.AddEvents(result.Events...)
blocking := actions.SendActionEvents(eventReceiver, result.Actions)
blockFailed := op.IsBlockFailed()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Limit block failure to the current WAF run

When _DD_APPSEC_BLOCKING_UNAVAILABLE is set, SetBlockFailed latches on the whole ContextOperation. Since the request is not interrupted in that mode, later RASP/WAF runs in the same request can still happen; copying op.IsBlockFailed() here marks those later runs as block:failure even when their own result.Actions did not request a block, so rasp.rule.match telemetry is misclassified for subsequent SQL/SSRF/etc. matches. Gate this value on the current blocking result or otherwise track the failure per run before passing it to RegisterWafRun.

Useful? React with 👍 / 👎.

@RomainMuller RomainMuller changed the title feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure (APPSEC-62578) Apr 30, 2026

@eliottness eliottness left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After careful reflexion all the "Blocking Failure" code paths will never trigger in production because the blockingUnavailable boolean that you use to trigger those was designed in the first place to not have to receiving blocking rules from remoteconfig by not sending the blocking capabilities.

Comment thread internal/appsec/listener/waf/waf.go Outdated
Comment on lines +128 to +131
if f.blockingUnavailable {
log.Debug("appsec: blocking event detected but blocking is unavailable")
op.SetBlockFailed()
} else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only happens in proxies. Not exactly meaningful as the target for "framework that don't allow us to block". What would really make sense in terms of "blocking failed" should be emitted in case we are trying to block when the customer already used ResponseWriter.WriteHeader, which we can do btw because we have a wrapper for it in APM code. At some point I had a PR to do this but this was just too complex and low prio to finish because it goes hand in hand with trying to delay the response from being sent until we know if we will not be able to block or the response payload become too big.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed entirely in 92353b6 — the whole block_failure feature (SetBlockFailed/IsBlockFailed, block_failure: telemetry tag, block:failure RASP variant, _dd.appsec.block.failed span tag) is dead code because blockingUnavailable suppresses blocking capability advertisements to RC so BlockingSecurityEvent can never fire.

Comment thread internal/appsec/emitter/waf/context.go Outdated
Comment on lines +48 to +49
// blockFailed is set when the WAF requested a block but the framework could not honour it.
blockFailed bool

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the metrics object just above, is this really necessary to add this boolean here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone in 92353b6 — the blockFailed bool field is removed along with the rest of the block_failure feature.

@RomainMuller RomainMuller changed the title feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure (APPSEC-62578) feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure Apr 30, 2026

@eliottness eliottness left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of RFC-1012 ASM Tags, Metrics & Logs Consolidation changes.

Comment thread internal/appsec/emitter/waf/metrics.go Outdated
m.exceptionType = errType
msg := err.Error()
if len(msg) > maxExceptionMsgBytes {
msg = string([]byte(msg)[:maxExceptionMsgBytes])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit — byte-slicing can split multi-byte UTF-8 characters

Slicing at maxExceptionMsgBytes (512) bytes can cut a multi-byte UTF-8 character mid-sequence, producing trailing invalid bytes. Not a correctness bug in Go (strings tolerate arbitrary bytes), but could cause issues if downstream consumers (backend, log pipelines) are strict about UTF-8.

If worth addressing, a common pattern is to walk backward to the last valid rune boundary, or apply strings.ToValidUTF8 after truncation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92353b6 — switched to strings.ToValidUTF8(msg[:maxExceptionMsgBytes], "") which drops any incomplete trailing rune rather than preserving the raw bytes.

Comment thread internal/appsec/listener/waf/waf.go Outdated
telemetryMetrics: telemetryMetrics,
metaStructAvailable: cfg.MetaStructAvailable,
rulesVersion: rulesVersion,
rcClientID: remoteconfig.ClientID(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational — remoteconfig.ClientID() is captured at Feature creation, not lazily

ClientID() returns "" if the RC client hasn't started yet. If the Feature is created before RC starts (possible during early startup), rcClientID is permanently empty for this Feature instance.

Likely fine since an RC config update would create a fresh Feature with the correct ID. But if the first WAF instance is expected to carry the ID (e.g., RC starts between Feature creation and first request), reading ClientID() lazily inside the reportRulesTags.Do closure would fix it:

waf.reportRulesTags.Do(func() {
    AddRulesMonitoringTags(op, remoteconfig.ClientID())
})

Is the current capture-at-creation behavior intentional?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92353b6 — moved the remoteconfig.ClientID() call inside the reportRulesTags.Do closure so it reads the ID lazily on first request rather than at Feature creation time.

@eliottness eliottness left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

x

m.logger.Error("unexpected call to RASPRuleTypeFromAddressSet", slog.Any("error", telemetrylog.NewSafeError(in)))
m.RecordException(ExceptionTypeInstrumentation, in)
}
m.raspError(in, ruleType)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low severity — fallthrough to raspError with zero-value rule type when !ok

When RASPRuleTypeFromAddressSet returns !ok, ruleType is the zero value (RASPRuleTypeLFI = 0). The instrumentation exception is recorded correctly, but execution falls through to raspError, which:

  1. Updates m.RASPErrorCodes[RASPRuleTypeLFI] — attributing the error to the wrong rule type.
  2. Emits a rasp.error telemetry metric tagged as LFI.

This was pre-existing (old code had the same fallthrough with m.logger.Error), but since this block was rewritten, adding a return after RecordException would be a natural fix:

if !ok {
    m.RecordException(ExceptionTypeInstrumentation, in)
    return
}
m.raspError(in, ruleType)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92353b6 — added return after RecordException in the !ok branch, also wrapped the error with the address-set context (same commit as 3167015464 below).

m.logger.Error("unexpected call to RASPRuleTypeFromAddressSet", slog.Any("error", telemetrylog.NewSafeError(in)))
m.RecordException(ExceptionTypeInstrumentation, in)
}
m.raspError(in, ruleType)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: missing return after failed RASP rule-type lookup — can index OOB

When RASPRuleTypeFromAddressSet returns ok==false, ruleType gets RASPRuleTypeLFI (zero value). The instrumentation exception is recorded, but execution falls through to m.raspError(in, ruleType) which attributes the error to the wrong rule type and emits rasp.error telemetry tagged as LFI.

Since this block was rewritten in this PR, adding a return is a natural fix:

if !ok {
    m.RecordException(ExceptionTypeInstrumentation,
        fmt.Errorf("unknown RASP rule type for scope %q: %w", addrs.TimerKey, in))
    return
}
m.raspError(in, ruleType)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92353b6 — added return after RecordException and wrapped the error with the address-set for context.

Comment thread internal/appsec/emitter/waf/metrics.go Outdated
Comment on lines +427 to +433
m.exceptionOnce.Do(func() {
m.exceptionType = errType
msg := err.Error()
if len(msg) > maxExceptionMsgBytes {
msg = string([]byte(msg)[:maxExceptionMsgBytes])
}
m.exceptionMsg = msg

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Race: sync.Once serializes the write, but reads of exceptionType/exceptionMsg are unprotected

ExceptionType() and ExceptionMsg() (lines 443-446) are plain field reads. AddWAFMonitoringTags calls them from onFinish, but in-flight Run() goroutines may still be inside RecordException. sync.Once.Do only guarantees that the func runs once — it does NOT create a happens-before edge for reads from a different goroutine that doesn't call Do.

This is a data race -race would flag, and can produce torn reads (type from one call, empty message).

Fix: publish both fields atomically via atomic.Pointer:

type recordedException struct{ typ, msg string }

// in ContextMetrics:
exception atomic.Pointer[recordedException]

Store inside Once.Do, load in ExceptionType()/ExceptionMsg().

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 91ab7c0 — replaced the two plain exceptionType/exceptionMsg string fields with atomic.Pointer[exceptionRecord]. The write happens inside Once.Do (still first-error-wins); the getters do an atomic Load, which gives a proper happens-before with any reader regardless of whether it called Do.

Comment thread internal/appsec/emitter/waf/metrics.go Outdated
m.exceptionType = errType
msg := err.Error()
if len(msg) > maxExceptionMsgBytes {
msg = string([]byte(msg)[:maxExceptionMsgBytes])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: byte slicing can produce invalid UTF-8 in _dd.appsec.error.message

string([]byte(msg)[:maxExceptionMsgBytes]) splits at a raw byte boundary, which can cut a multi-byte rune. Also allocates twice (string→[]byte→string).

msg = msg[:maxExceptionMsgBytes]
for !utf8.ValidString(msg) {
    msg = msg[:len(msg)-1]
}

(needs "unicode/utf8" import)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92353b6 — same as the other UTF-8 nit; using strings.ToValidUTF8.

Comment thread internal/appsec/emitter/waf/metrics.go Outdated
m.wafError(in)
default:
m.logger.Error("unexpected scope name", slog.String("scope", string(addrs.TimerKey)))
m.RecordException(ExceptionTypeInstrumentation, in)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context loss: invalid scope value dropped from telemetry

The previous code logged slog.String("scope", string(addrs.TimerKey)). Now only the bare in error reaches RecordException, so production telemetry will say "appsec exception" without identifying which scope triggered it.

m.RecordException(ExceptionTypeInstrumentation,
    fmt.Errorf("unexpected WAF error scope %q: %w", addrs.TimerKey, in))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 91ab7c0 — the default case now wraps the error: fmt.Errorf("unexpected WAF error scope %q: %w", addrs.TimerKey, in).

Comment on lines +105 to +108

if feature.APISec.Enabled && op.Route() == "" {
telemetry.Count(telemetry.NamespaceAppSec, "api_security.missing_route", []string{"framework:" + op.Framework()}).Submit(1)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI: this metric breaks test_shema_metric[gin] system test

The system test asserts that only api_security.request.schema and api_security.request.no_schema telemetry metrics are present. This new api_security.missing_route metric fails that assertion:

AssertionError: Only api_security.request.schema metrics should be present,
no missing routes should be generated

The system test allowlist needs updating to include api_security.missing_route.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92353b6 — added && resp.StatusCode != 404 guard. A 404 means no route matched (expected gin behaviour); only a non-404 response with an empty route indicates the framework genuinely lacks routing support. Matches dd-trace-py behaviour.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 3b54935 — the previous != 404 guard wasn't sufficient: gin also returns an empty route for trailing-slash redirects (301), method-not-allowed (405), and any other response without a matched handler. Changed the condition to status >= 200 && status < 300: only 2xx responses are meaningful for "missing route" diagnostics (a framework without routing support still exposes this on successful requests; gin never triggers it because all its 2xx responses carry a non-empty route).

@pr-commenter

pr-commenter Bot commented Apr 30, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-04-30 11:25:54

Comparing candidate commit 3b54935 in PR branch romain.marcadier/APPSEC-62578/asm-rfc1012-consolidation with baseline commit 997e8b2 in branch main.

Found 3 performance improvements and 0 performance regressions! Performance is the same for 266 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:BenchmarkOTLPProtoSize/1000spans-25

  • 🟩 execution_time [-8.399µs; -6.567µs] or [-2.672%; -2.089%]

scenario:BenchmarkOTLPProtoSize/100spans-25

  • 🟩 execution_time [-858.391ns; -744.009ns] or [-2.725%; -2.362%]

scenario:BenchmarkOTLPProtoSize/10spans-25

  • 🟩 execution_time [-96.365ns; -77.035ns] or [-2.978%; -2.380%]

Remove the `block_failure` feature entirely. `blockingUnavailable` was
designed to suppress blocking capability advertisements to RC, so the
backend never sends blocking rules and `BlockingSecurityEvent` can
never fire. The `SetBlockFailed`/`IsBlockFailed` path, the
`block_failure:` tag on `waf.requests`, the `block:failure` variant of
`rasp.rule.match`, and the `_dd.appsec.block.failed` span tag were all
unreachable dead code.

Fix a fallthrough bug in `IncWafError`: when
`RASPRuleTypeFromAddressSet` returns `!ok`, execution fell through to
`raspError` with the zero value of `RASPRuleType` (LFI), misattributing
the error code and telemetry metric to the wrong rule type. Add an
explicit `return` after `RecordException` in that branch.

Fix UTF-8 safety in `RecordException`: truncating the error message
with a raw byte slice at 512 bytes could split a multi-byte rune,
producing an ill-formed string. Use `strings.ToValidUTF8` instead.

Read `remoteconfig.ClientID()` lazily inside the `reportRulesTags.Do`
closure. The previous code captured the ID at `Feature` creation time;
if RC had not started yet at that point the field would permanently
hold an empty string for that `Feature` instance.

Exclude HTTP 404 responses from the `api_security.missing_route`
metric. A 404 means no route matched the request — the framework does
provide routing support; this particular URL simply has no handler.
Emitting the metric for 404s was causing false positives in frameworks
like gin that correctly expose routes for all matched paths. This
aligns with the behaviour of dd-trace-py.

JJ-Change-Id: snqvtx
Fix a data race in ContextMetrics: ExceptionType() and ExceptionMsg()
were plain field reads with no synchronisation against concurrent
RecordException() calls from in-flight Run() goroutines. sync.Once
only serialises the write inside Do — it does not establish a
happens-before for goroutines that read the fields without calling Do.
Replace the two plain string fields with an atomic.Pointer[exceptionRecord]
so both the store (inside Once.Do) and the load (in the getters) are
race-free.

Add error context to the two RecordException call-sites in IncWafError
that previously forwarded the raw error without any scope information:
the unknown-RASP-rule-type branch now wraps the error with the address
set, and the default/unexpected-scope branch wraps it with the timer
key, making production telemetry actionable.

JJ-Change-Id: zzpozt
Comment thread internal/appsec/emitter/waf/metrics.go Dismissed
Gin (and other routing-capable frameworks) return empty route for
requests that do not match any handler: trailing-slash redirects (301),
no-match 404s, method-not-allowed 405s, etc. The previous guard only
excluded 404, leaving 3xx redirects able to trigger the metric.

Restrict the emission to 2xx status codes. A framework that genuinely
lacks routing support will still trigger the metric for its successful
responses; frameworks like gin will never trigger it because all their
2xx responses carry a non-empty route.

JJ-Change-Id: kztzrl
@RomainMuller
RomainMuller requested a review from eliottness April 30, 2026 10:16

@eliottness eliottness left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All issues from the first review pass are resolved. No new bugs found in the current diff. LGTM.

@RomainMuller
RomainMuller enabled auto-merge (squash) April 30, 2026 16:00

@genesor genesor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

One question about block_failure, The PR description lists block_failure work (new _dd.appsec.block.failed=1 span tag, waf.requests{block_failure:bool}, and block:failure variant on rasp.rule.match), but the diff only deletes the related TODOs without adding the logic.

Does this mean that the feature is already implemented and the TODOs were not removed ?

Not blocking but would be nice to get a reply once you're back @RomainMuller

@RomainMuller
RomainMuller merged commit 625c602 into main May 4, 2026
213 of 215 checks passed
@RomainMuller
RomainMuller deleted the romain.marcadier/APPSEC-62578/asm-rfc1012-consolidation branch May 4, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants