feat(appsec): RFC-1012 ASM Tags, Metrics & Logs Consolidation gap closure#4708
Conversation
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0hEQgyKP54kM1LZU98GL8MaP60tQrA/): the span tag \`_dd.rc.client_id\` must be set once per WAF instance on the service entry span so the backend can correlate WAF decisions with the Remote Configuration client that delivered the rule-set. - Add \`ClientID() string\` to \`internal/remoteconfig\` — returns the singleton client's ID (empty when RC is not started). - Store \`rcClientID\` on the WAF \`Feature\` struct at construction time (captured via \`remoteconfig.ClientID()\`). - Pass \`rcClientID\` to \`AddRulesMonitoringTags\`, which now emits the \`_dd.rc.client_id\` tag when the value is non-empty. The tag is emitted exactly once per WAF instance via the existing \`sync.Once\` guard. JJ-Change-Id: slsvrr
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h EQgyKP54kM1LZU98GL8MaP60tQrA/): two breaking changes: 1. \`_dd.appsec.waf.error\` and \`_dd.appsec.rasp.error\` span tags were emitting a cumulative **count** of errors. The RFC requires the closest-to-zero (least-negative) ddwaf_run error code. Replace the \`atomic.Uint32\` count fields with \`atomic.Int32\` code accumulators (sentinel 0 = no error; updated via CAS to keep the max value, which for negative codes is the closest to zero). For RASP the accumulator is per rule type; the span tag reports the closest-to-zero across all rule types. 2. The telemetry metrics \`appsec.waf.error\` and \`appsec.rasp.error\` had a tag \`error_code:<n>\`. The RFC mandates \`waf_error:<n>\`. Hard rename applied to both emitters in \`metrics.go\`. Unit tests added for the CAS accumulator and for the span tag behaviour (both WAF and RASP paths). JJ-Change-Id: typulw
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h EQgyKP54kM1LZU98GL8MaP60tQrA/): when the WAF requests a block but the framework cannot honour it (\`BlockingUnavailable=true\`), the library must record the failure rather than falsely claiming a successful block. Changes: - \`RequestMilestones\` gains \`blockFailed bool\`; the WAF-scope accumulator propagates it so \`waf.requests\` includes a \`block_failure:<bool>\` tag. - \`ContextOperation\` gets \`SetBlockFailed()\` / \`IsBlockFailed()\`; the blocking event handler in \`SetupActionHandlers\` calls \`SetBlockFailed()\` instead of setting \`appsec.blocked\` when \`blockingUnavailable=true\`. - \`run.go\`: milestones now set \`requestBlocked: blocking && !blockFailed\` so the two flags are mutually exclusive. - \`AddWAFMonitoringTags\`: emits \`_dd.appsec.block.failed=1\` when set. - RASP \`rasp.rule.match\` block tag extended with \`block:failure\` case. - \`Feature\` stores \`blockingUnavailable\` from \`cfg.BlockingUnavailable\`. - Existing \`waf.requests\` tag assertions updated to include the new tag. JJ-Change-Id: nwvzpx
…tags Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h EQgyKP54kM1LZU98GL8MaP60tQrA/): when a WAF or RASP run produces a non-timeout error, record the exception category and a truncated error message as root span tags so the backend can surface them. Changes: - \`ContextMetrics.RecordException(errType, err)\` stores the first error per request (first-error-wins via \`sync.Once\`) and emits a telemetry log with the RFC-1012 \`log_type\` tag and a stack trace. - Exception type constants (\`ExceptionTypeWAF\`, \`ExceptionTypeRASP\`, \`ExceptionTypeInstrumentation\`) exported from \`emitter/waf\`. - \`IncWafError\` calls \`RecordException\` with the appropriate type (WAF vs RASP vs instrumentation path). - \`AddWAFMonitoringTags\` emits \`_dd.appsec.error.type\` and \`_dd.appsec.error.message\` (≤512 bytes) when an exception was recorded. - \`NewWAFFeature\`'s libddwaf load warning tagged with \`log_type:appsec::waf::exception\` + stack trace. - Round-tripper body parse warnings tagged with \`log_type:appsec::instrumentation::exception\` + stack trace. JJ-Change-Id: vpwkuq
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h EQgyKP54kM1LZU98GL8MaP60tQrA/): two log-type renames and two new exception-typed log sites. - \`wafmanager.go\`: rename ad-hoc \`log_type:local::diagnostic\` to the RFC-mandated \`appsec::waf::ruleset::diagnostic\` on the local-rules diagnostic path. - \`internal/appsec/remoteconfig.go\`: the two existing bare \`telemetrylog.Error\` calls in RC subscriber error paths now carry \`log_type:rc::asm::exception\` and \`WithStacktrace()\` as required by RFC-1012 for exception-typed telemetry logs. No import cycle: \`internal/remoteconfig\` does not depend on \`internal/telemetry/log\`, verified via \`go list -deps\`. JJ-Change-Id: lumlzm
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h EQgyKP54kM1LZU98GL8MaP60tQrA/): three declared-but-never-emitted metrics are now populated. \`appsec.api_security.missing_route\`: - Emitted in \`httpsec.Feature.OnResponse\` when API security is enabled but \`op.Route()\` is empty (framework did not set the matched route). - Tagged with \`framework:<name>\`. \`appsec.instrum.user_auth.missing_user_login\`: - Emitted from \`usersec.Feature.OnFinish\` when the login field is empty for \`login_success\` or \`login_failure\` events. \`appsec.instrum.user_auth.missing_user_id\`: - Emitted from \`usersec.Feature.OnFinish\` when the user-ID field is empty for \`login_success\` or \`authenticated_request\` events. Both user-auth metrics carry \`event_type:<value>\` and \`framework:<name>\` tags. The framework is resolved by walking the operation parent chain to find the nearest \`*httpsec.HandlerOperation\`; falls back to "unknown". JJ-Change-Id: ryrtoo
Implements RFC-1012 (https://docs.google.com/document/d/1D4hkC0jwwUyeo0h EQgyKP54kM1LZU98GL8MaP60tQrA/): the previously declared-but-never-emitted \`rasp.rule.skipped\` count metric is now populated. \`rasp.rule.skipped\` (Count): - A lazily-filled map keyed by (rule_type, reason) is added to \`HandleMetrics\`. - \`ContextMetrics.SkipRASPRule(ruleType, reason)\` is the public entry point; it emits one count per skip event. - \`ContextOperation.Run\` emits \`reason:after-request\` when the WAF context has already been closed at the time of a RASP call. Note: \`rasp.rule.duration\` is not implemented here — it is absent from DataDog/dd-go's common_metrics.json and therefore not yet in the telemetry spec. It will be added in a follow-up once the spec is updated. JJ-Change-Id: tmqsym
Merges all RFC-1012 sibling PRs to verify zero integration conflicts and a clean combined test run. Not intended for submission. Parents: romain.marcadier/asm-rc-client-id romain.marcadier/asm-waf-rasp-error-semantics romain.marcadier/asm-block-failure romain.marcadier/asm-error-type-message romain.marcadier/asm-log-type-vocab romain.marcadier/asm-rasp-rule-duration-skipped romain.marcadier/asm-route-and-user-auth-metrics Conflict resolution: hot-spot files touched by multiple PRs (metrics.go, tags.go, tags_test.go, waf.go) were merged by hand, taking all additions from every sibling branch. JJ-Change-Id: kwvkxp
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 3b54935 | Docs | Datadog PR Page | Give us feedback! |
JJ-Change-Id: splmkw
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e004320f44
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| wafTimeout := errors.Is(err, waferrors.ErrTimeout) | ||
| rateLimited := op.AddEvents(result.Events...) | ||
| blocking := actions.SendActionEvents(eventReceiver, result.Actions) | ||
| blockFailed := op.IsBlockFailed() |
There was a problem hiding this comment.
Limit block failure to the current WAF run
When _DD_APPSEC_BLOCKING_UNAVAILABLE is set, SetBlockFailed latches on the whole ContextOperation. Since the request is not interrupted in that mode, later RASP/WAF runs in the same request can still happen; copying op.IsBlockFailed() here marks those later runs as block:failure even when their own result.Actions did not request a block, so rasp.rule.match telemetry is misclassified for subsequent SQL/SSRF/etc. matches. Gate this value on the current blocking result or otherwise track the failure per run before passing it to RegisterWafRun.
Useful? React with 👍 / 👎.
eliottness
left a comment
There was a problem hiding this comment.
After careful reflexion all the "Blocking Failure" code paths will never trigger in production because the blockingUnavailable boolean that you use to trigger those was designed in the first place to not have to receiving blocking rules from remoteconfig by not sending the blocking capabilities.
| if f.blockingUnavailable { | ||
| log.Debug("appsec: blocking event detected but blocking is unavailable") | ||
| op.SetBlockFailed() | ||
| } else { |
There was a problem hiding this comment.
This only happens in proxies. Not exactly meaningful as the target for "framework that don't allow us to block". What would really make sense in terms of "blocking failed" should be emitted in case we are trying to block when the customer already used ResponseWriter.WriteHeader, which we can do btw because we have a wrapper for it in APM code. At some point I had a PR to do this but this was just too complex and low prio to finish because it goes hand in hand with trying to delay the response from being sent until we know if we will not be able to block or the response payload become too big.
There was a problem hiding this comment.
Removed entirely in 92353b6 — the whole block_failure feature (SetBlockFailed/IsBlockFailed, block_failure: telemetry tag, block:failure RASP variant, _dd.appsec.block.failed span tag) is dead code because blockingUnavailable suppresses blocking capability advertisements to RC so BlockingSecurityEvent can never fire.
| // blockFailed is set when the WAF requested a block but the framework could not honour it. | ||
| blockFailed bool |
There was a problem hiding this comment.
I see the metrics object just above, is this really necessary to add this boolean here?
There was a problem hiding this comment.
Gone in 92353b6 — the blockFailed bool field is removed along with the rest of the block_failure feature.
eliottness
left a comment
There was a problem hiding this comment.
Review of RFC-1012 ASM Tags, Metrics & Logs Consolidation changes.
| m.exceptionType = errType | ||
| msg := err.Error() | ||
| if len(msg) > maxExceptionMsgBytes { | ||
| msg = string([]byte(msg)[:maxExceptionMsgBytes]) |
There was a problem hiding this comment.
Nit — byte-slicing can split multi-byte UTF-8 characters
Slicing at maxExceptionMsgBytes (512) bytes can cut a multi-byte UTF-8 character mid-sequence, producing trailing invalid bytes. Not a correctness bug in Go (strings tolerate arbitrary bytes), but could cause issues if downstream consumers (backend, log pipelines) are strict about UTF-8.
If worth addressing, a common pattern is to walk backward to the last valid rune boundary, or apply strings.ToValidUTF8 after truncation.
There was a problem hiding this comment.
Fixed in 92353b6 — switched to strings.ToValidUTF8(msg[:maxExceptionMsgBytes], "") which drops any incomplete trailing rune rather than preserving the raw bytes.
| telemetryMetrics: telemetryMetrics, | ||
| metaStructAvailable: cfg.MetaStructAvailable, | ||
| rulesVersion: rulesVersion, | ||
| rcClientID: remoteconfig.ClientID(), |
There was a problem hiding this comment.
Informational — remoteconfig.ClientID() is captured at Feature creation, not lazily
ClientID() returns "" if the RC client hasn't started yet. If the Feature is created before RC starts (possible during early startup), rcClientID is permanently empty for this Feature instance.
Likely fine since an RC config update would create a fresh Feature with the correct ID. But if the first WAF instance is expected to carry the ID (e.g., RC starts between Feature creation and first request), reading ClientID() lazily inside the reportRulesTags.Do closure would fix it:
waf.reportRulesTags.Do(func() {
AddRulesMonitoringTags(op, remoteconfig.ClientID())
})Is the current capture-at-creation behavior intentional?
There was a problem hiding this comment.
Fixed in 92353b6 — moved the remoteconfig.ClientID() call inside the reportRulesTags.Do closure so it reads the ID lazily on first request rather than at Feature creation time.
| m.logger.Error("unexpected call to RASPRuleTypeFromAddressSet", slog.Any("error", telemetrylog.NewSafeError(in))) | ||
| m.RecordException(ExceptionTypeInstrumentation, in) | ||
| } | ||
| m.raspError(in, ruleType) |
There was a problem hiding this comment.
Low severity — fallthrough to raspError with zero-value rule type when !ok
When RASPRuleTypeFromAddressSet returns !ok, ruleType is the zero value (RASPRuleTypeLFI = 0). The instrumentation exception is recorded correctly, but execution falls through to raspError, which:
- Updates
m.RASPErrorCodes[RASPRuleTypeLFI]— attributing the error to the wrong rule type. - Emits a
rasp.errortelemetry metric tagged as LFI.
This was pre-existing (old code had the same fallthrough with m.logger.Error), but since this block was rewritten, adding a return after RecordException would be a natural fix:
if !ok {
m.RecordException(ExceptionTypeInstrumentation, in)
return
}
m.raspError(in, ruleType)There was a problem hiding this comment.
Fixed in 92353b6 — added return after RecordException in the !ok branch, also wrapped the error with the address-set context (same commit as 3167015464 below).
| m.logger.Error("unexpected call to RASPRuleTypeFromAddressSet", slog.Any("error", telemetrylog.NewSafeError(in))) | ||
| m.RecordException(ExceptionTypeInstrumentation, in) | ||
| } | ||
| m.raspError(in, ruleType) |
There was a problem hiding this comment.
Bug: missing return after failed RASP rule-type lookup — can index OOB
When RASPRuleTypeFromAddressSet returns ok==false, ruleType gets RASPRuleTypeLFI (zero value). The instrumentation exception is recorded, but execution falls through to m.raspError(in, ruleType) which attributes the error to the wrong rule type and emits rasp.error telemetry tagged as LFI.
Since this block was rewritten in this PR, adding a return is a natural fix:
if !ok {
m.RecordException(ExceptionTypeInstrumentation,
fmt.Errorf("unknown RASP rule type for scope %q: %w", addrs.TimerKey, in))
return
}
m.raspError(in, ruleType)There was a problem hiding this comment.
Fixed in 92353b6 — added return after RecordException and wrapped the error with the address-set for context.
| m.exceptionOnce.Do(func() { | ||
| m.exceptionType = errType | ||
| msg := err.Error() | ||
| if len(msg) > maxExceptionMsgBytes { | ||
| msg = string([]byte(msg)[:maxExceptionMsgBytes]) | ||
| } | ||
| m.exceptionMsg = msg |
There was a problem hiding this comment.
Race: sync.Once serializes the write, but reads of exceptionType/exceptionMsg are unprotected
ExceptionType() and ExceptionMsg() (lines 443-446) are plain field reads. AddWAFMonitoringTags calls them from onFinish, but in-flight Run() goroutines may still be inside RecordException. sync.Once.Do only guarantees that the func runs once — it does NOT create a happens-before edge for reads from a different goroutine that doesn't call Do.
This is a data race -race would flag, and can produce torn reads (type from one call, empty message).
Fix: publish both fields atomically via atomic.Pointer:
type recordedException struct{ typ, msg string }
// in ContextMetrics:
exception atomic.Pointer[recordedException]Store inside Once.Do, load in ExceptionType()/ExceptionMsg().
There was a problem hiding this comment.
Fixed in 91ab7c0 — replaced the two plain exceptionType/exceptionMsg string fields with atomic.Pointer[exceptionRecord]. The write happens inside Once.Do (still first-error-wins); the getters do an atomic Load, which gives a proper happens-before with any reader regardless of whether it called Do.
| m.exceptionType = errType | ||
| msg := err.Error() | ||
| if len(msg) > maxExceptionMsgBytes { | ||
| msg = string([]byte(msg)[:maxExceptionMsgBytes]) |
There was a problem hiding this comment.
Nit: byte slicing can produce invalid UTF-8 in _dd.appsec.error.message
string([]byte(msg)[:maxExceptionMsgBytes]) splits at a raw byte boundary, which can cut a multi-byte rune. Also allocates twice (string→[]byte→string).
msg = msg[:maxExceptionMsgBytes]
for !utf8.ValidString(msg) {
msg = msg[:len(msg)-1]
}(needs "unicode/utf8" import)
There was a problem hiding this comment.
Fixed in 92353b6 — same as the other UTF-8 nit; using strings.ToValidUTF8.
| m.wafError(in) | ||
| default: | ||
| m.logger.Error("unexpected scope name", slog.String("scope", string(addrs.TimerKey))) | ||
| m.RecordException(ExceptionTypeInstrumentation, in) |
There was a problem hiding this comment.
Context loss: invalid scope value dropped from telemetry
The previous code logged slog.String("scope", string(addrs.TimerKey)). Now only the bare in error reaches RecordException, so production telemetry will say "appsec exception" without identifying which scope triggered it.
m.RecordException(ExceptionTypeInstrumentation,
fmt.Errorf("unexpected WAF error scope %q: %w", addrs.TimerKey, in))There was a problem hiding this comment.
Fixed in 91ab7c0 — the default case now wraps the error: fmt.Errorf("unexpected WAF error scope %q: %w", addrs.TimerKey, in).
|
|
||
| if feature.APISec.Enabled && op.Route() == "" { | ||
| telemetry.Count(telemetry.NamespaceAppSec, "api_security.missing_route", []string{"framework:" + op.Framework()}).Submit(1) | ||
| } |
There was a problem hiding this comment.
CI: this metric breaks test_shema_metric[gin] system test
The system test asserts that only api_security.request.schema and api_security.request.no_schema telemetry metrics are present. This new api_security.missing_route metric fails that assertion:
AssertionError: Only api_security.request.schema metrics should be present,
no missing routes should be generated
The system test allowlist needs updating to include api_security.missing_route.
There was a problem hiding this comment.
Fixed in 92353b6 — added && resp.StatusCode != 404 guard. A 404 means no route matched (expected gin behaviour); only a non-404 response with an empty route indicates the framework genuinely lacks routing support. Matches dd-trace-py behaviour.
There was a problem hiding this comment.
Updated in 3b54935 — the previous != 404 guard wasn't sufficient: gin also returns an empty route for trailing-slash redirects (301), method-not-allowed (405), and any other response without a matched handler. Changed the condition to status >= 200 && status < 300: only 2xx responses are meaningful for "missing route" diagnostics (a framework without routing support still exposes this on successful requests; gin never triggers it because all its 2xx responses carry a non-empty route).
BenchmarksBenchmark execution time: 2026-04-30 11:25:54 Comparing candidate commit 3b54935 in PR branch Found 3 performance improvements and 0 performance regressions! Performance is the same for 266 metrics, 9 unstable metrics.
|
Remove the `block_failure` feature entirely. `blockingUnavailable` was designed to suppress blocking capability advertisements to RC, so the backend never sends blocking rules and `BlockingSecurityEvent` can never fire. The `SetBlockFailed`/`IsBlockFailed` path, the `block_failure:` tag on `waf.requests`, the `block:failure` variant of `rasp.rule.match`, and the `_dd.appsec.block.failed` span tag were all unreachable dead code. Fix a fallthrough bug in `IncWafError`: when `RASPRuleTypeFromAddressSet` returns `!ok`, execution fell through to `raspError` with the zero value of `RASPRuleType` (LFI), misattributing the error code and telemetry metric to the wrong rule type. Add an explicit `return` after `RecordException` in that branch. Fix UTF-8 safety in `RecordException`: truncating the error message with a raw byte slice at 512 bytes could split a multi-byte rune, producing an ill-formed string. Use `strings.ToValidUTF8` instead. Read `remoteconfig.ClientID()` lazily inside the `reportRulesTags.Do` closure. The previous code captured the ID at `Feature` creation time; if RC had not started yet at that point the field would permanently hold an empty string for that `Feature` instance. Exclude HTTP 404 responses from the `api_security.missing_route` metric. A 404 means no route matched the request — the framework does provide routing support; this particular URL simply has no handler. Emitting the metric for 404s was causing false positives in frameworks like gin that correctly expose routes for all matched paths. This aligns with the behaviour of dd-trace-py. JJ-Change-Id: snqvtx
Fix a data race in ContextMetrics: ExceptionType() and ExceptionMsg() were plain field reads with no synchronisation against concurrent RecordException() calls from in-flight Run() goroutines. sync.Once only serialises the write inside Do — it does not establish a happens-before for goroutines that read the fields without calling Do. Replace the two plain string fields with an atomic.Pointer[exceptionRecord] so both the store (inside Once.Do) and the load (in the getters) are race-free. Add error context to the two RecordException call-sites in IncWafError that previously forwarded the raw error without any scope information: the unknown-RASP-rule-type branch now wraps the error with the address set, and the default/unexpected-scope branch wraps it with the timer key, making production telemetry actionable. JJ-Change-Id: zzpozt
Gin (and other routing-capable frameworks) return empty route for requests that do not match any handler: trailing-slash redirects (301), no-match 404s, method-not-allowed 405s, etc. The previous guard only excluded 404, leaving 3xx redirects able to trigger the metric. Restrict the emission to 2xx status codes. A framework that genuinely lacks routing support will still trigger the metric for its successful responses; frameworks like gin will never trigger it because all their 2xx responses carry a non-empty route. JJ-Change-Id: kztzrl
eliottness
left a comment
There was a problem hiding this comment.
All issues from the first review pass are resolved. No new bugs found in the current diff. LGTM.
genesor
left a comment
There was a problem hiding this comment.
LGTM!
One question about block_failure, The PR description lists block_failure work (new _dd.appsec.block.failed=1 span tag, waf.requests{block_failure:bool}, and block:failure variant on rasp.rule.match), but the diff only deletes the related TODOs without adding the logic.
Does this mean that the feature is already implemented and the TODOs were not removed ?
Not blocking but would be nice to get a reply once you're back @RomainMuller
Summary
Closes the remaining MUST-implement gaps from RFC-1012 — ASM Tags, Metrics & Logs Consolidation in
dd-trace-go.Changes
_dd.rc.client_idremoteconfig.ClientID()waf/rasp.errorsemantics_dd.appsec.waf.error/rasp.errornow emit the closest-to-zero ddwaf_run error code instead of a count; telemetry tag renamederror_code:→waf_error:(breaking)block_failure_dd.appsec.block.failed=1span tag,waf.requests{block_failure:bool}telemetry tag, andrasp.rule.match{block:failure}variant whenBlockingUnavailable=trueerror.type/message_dd.appsec.error.typeand_dd.appsec.error.message(≤512 B) set on root span for first WAF/RASP exception per request; all exception log sites now carry RFC-1012log_typetag + stack tracelog_typevocablocal::diagnostic→appsec::waf::ruleset::diagnostic; RC subscriber error paths taggedrc::asm::exceptionrasp.rule.skippedreason:after-requestwhen a RASP call arrives after the WAF context is closedapi_security.missing_route+ user-auth metricsappsec.api_security.missing_route,instrum.user_auth.missing_user_login,instrum.user_auth.missing_user_idnow emittedOut of scope (per RFC or blocked)
_dd.appsec.excluded/request_excluded— blocked on libddwafsdk.event{signup}— needs user-lifecycle RFC owner sign-off before new SDK surfacerasp.rule.duration— not yet registered inDataDog/dd-gocommon_metrics.json