feat(asm): add normalized HTTP route span tag for Flask#18343
Conversation
7330a2d to
4f8fa26
Compare
Codeowners resolved as |
|
BenchmarksBenchmark execution time: 2026-06-01 10:47:23 Comparing candidate commit 6e959aa in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 83 metrics, 0 unstable metrics. scenario:iastaspectsospath-ospathbasename_aspect
|
185e209 to
0e1ec61
Compare
Extends RFC-1103 ``_dd.appsec.normalized_route`` to Flask, mirroring the FastAPI/Starlette (#17920) and Django (#18209) implementations. - ``ddtrace/appsec/_api_security/_normalized_route.py``: adds ``normalize_route_flask`` with a fast path (single-converter params, no catch-all) and a slow path covering multi-param-in-segment, ``<path:name>`` catch-all, ``any(...)`` converters, and static URL-encoding. - ``ddtrace/appsec/_handlers.py``: adds ``"flask"`` to ``_NORMALIZED_ROUTE_BY_INTEGRATION``; resolves the DispatcherMiddleware mount prefix from ``flask.resource.full`` so sub-app routes include their full assembled path. - Test app (flat + subapp): adds ``/multi-param/<first>.<last>/`` and ``/files/<path:file_path>`` routes. - ``tests/appsec/contrib_appsec/utils.py``: removes Flask from ``xfail_interface`` on the two normalized-route tests. - Unit tests: 50 new parametrized cases for ``normalize_route_flask``, fast-path eligibility, and invalid-input rejection. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Apply fixes from three rounds of dual review (Claude subagent + Codex):
_normalized_route.py
- _FLASK_PARAM_REGEX: change [^>:]+ to [^>]+ so converter arguments
containing `:` (e.g. custom <regex("[a:b]+"): slug>) are handled via
backtracking rather than stopping at the first `:`. Add clarifying comment.
- _normalize_route_flask_slow: document why non-terminal <path:...> segments
return None — Werkzeug allows them at runtime but RFC-1103 rule 5 mandates
catch-alls be terminal; omit-rather-than-guess is the safe contract.
- Add comment on catch-all-wins-segment behaviour for mixed path+non-path
params (Werkzeug rejects these at registration, so unreachable in practice).
_handlers.py
- Import FLASK_RESOURCE_FULL from ddtrace.internal.constants instead of
duplicating the string literal; removes the private _FLASK_RESOURCE_FULL_TAG
alias so the constant is used directly at the call site.
test_normalized_route.py
- Move normalize_route_flask import to file top; remove noqa E402.
- Add assert normalize_route_flask(route) == expected to fast-path test so
the public entry-point wiring is covered, not just the cached helper.
- Add test cases: param named <path> (no converter, not a catch-all),
custom converter with : in argument string.
- Add test_normalize_route_flask_dm_assembly_via_handler: isolated unit test
for the handler-level FLASK_RESOURCE_FULL assembly logic via mocks; use the
imported FLASK_RESOURCE_FULL constant rather than a bare string literal.
- Document why /<path:tail>/and-after returns None (RFC-1103 design choice,
not a Werkzeug constraint).
utils.py
- Add /multi-param/john.doe (no trailing slash) to test_normalized_route
parametrize to cover the non-trailing-slash route registration.
- Document why Flask is kept in xfail for the request_span_name-override
test (Flask middleware uses a fixed _request_call_name class attribute).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Three distinct CI failures on the PR: 1. Django/FastAPI appsec_threats tests: `test_normalized_route` was parametrized with `/multi-param/john.doe` (no trailing slash) added in a prior review cycle. Django and FastAPI only register the trailing-slash variant of that route, causing HTTP 301/308 redirects and assertion failures. Remove the no-trailing-slash case from the shared parametrize — the trailing-slash variant already exercises Flask multi-param normalization end-to-end, and unit tests in test_normalized_route.py cover both shapes. 2. Flask appsec_threats endpoint_discovery test: `endpoint_path_to_uri` only substituted `<int:name>` and `<str:name>` patterns. The new `/multi-param/<first>.<last>/` route uses bare `<name>` converters (no type prefix), causing the raw pattern to be used as the request URI, which returns 308 (redirect to trailing-slash variant). Add substitution rules for bare `<name>` and `<path:name>` catch-all, and add the two new routes to ENDPOINT_DISCOVERY_EXPECTED_PATHS. 3. Flask contrib snapshot tests: Flask now emits `_dd.appsec.normalized_route` for matched routes when API Security is active. The appsec snapshot tests (processexec_*, userblock_*) run with DD_APPSEC_ENABLED=true and reach matched routes, so the new tag now appears in received spans. Update all 12 affected snapshot files (both `_220` and legacy variants). The ipblock tests are not affected because they are blocked before route matching and do not emit the tag. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Two remaining failures after the previous fix: 1. Snapshot JSON trailing comma: `fmt-snapshots` introduced a trailing comma in the `meta` object of two `test_flask_processexec_osspawn` snapshot files, causing a `400: Illegal trailing comma before end of object` parse error in the test agent. Strip the trailing comma. 2. Endpoint discovery 308 redirect: registering both `/multi-param/<first>.<last>/` (trailing slash) and `/multi-param/<first>.<last>` (no trailing slash) causes the endpoint collection to discover both paths. `endpoint_path_to_uri` converts the bare `<name>` converters to `test`, yielding `/multi-param/test.test`, which Werkzeug redirects (308) to the trailing-slash variant — failing the 200/401 assertion. Remove the no-trailing-slash multi-param route from the Flask test apps (only the trailing-slash variant is needed for normalisation coverage). Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
0e1ec61 to
0b5a52b
Compare
The two osspawn snapshot files were formatted with Python's json.dump (4-space indentation) instead of the test-agent canonical format (2-space, compact). Run scripts/lint fmt-snapshots to restore the correct format so the prechecks snapshot-format check passes. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
- Remove duplicate module-level DM comment from _handlers.py (covered by inline)
- Shorten multi-line comments in _handlers.py and _normalized_route.py to one-liners
- Add ("/files/<path>", "/files/{path}") to test_normalize_route_flask_fast_path
to document that a param named "path" (no converter) takes the fast path,
distinct from <path:name> which is the catch-all converter
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
@codex review |
|
Codex Review: Didn't find any major issues. Swish! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
4b84f68
into
main
## Description Extends RFC-1103 `_dd.appsec.normalized_route` to Tornado, following the FastAPI/Starlette (#17920), Django (#18209), and Flask (#18343) implementations. Tornado's ddtrace integration produces `http.route` strings with `%s` as a placeholder for every capturing group (via `_regex_to_route`). The new normalizer maps those placeholders to parameter names sourced from `path_params`: a dict for named groups (`(?P<name>...)`), a list for positional groups, or `{}` for static routes. Optional trailing-slash patterns (`/?`) are treated as not declaring a trailing slash per RFC-1103 rule 1, consistent with the Django `^asm/?$` → `/asm` convention. **Changes:** - **`_normalized_route.py`** — adds `_normalize_route_tornado_cached` (LRU-cached, keyed on route + param-names tuple) and `normalize_route_tornado` (public entry point). Handles `%s` placeholder mapping, `/?` stripping, multi-param-in-segment combining with `+` (rule 5), and static-segment URL-encoding (rule 3). - **`_handlers.py`** — registers `"tornado": normalize_route_tornado` in `_NORMALIZED_ROUTE_BY_INTEGRATION`. - **`tornado_app/app.py`** — converts the `/asm/` route from positional to named capturing groups so `path_params` arrives as a dict with proper parameter names; adds `MultiParamHandler` (`/multi-param/`) and `FilesHandler` (`/files/`) for integration test coverage. - **`test_tornado.py`** — overrides `test_normalized_route` and `test_normalized_route_disabled_when_api_security_off` with Tornado-specific expected values (no trailing slash for `/?` routes); expands `ENDPOINT_DISCOVERY_EXPECTED_PATHS`. - **`test_normalized_route.py`** — 27 new unit tests for `normalize_route_tornado` covering all code paths. - **`utils.py`** — documents why Tornado is skipped in `test_normalized_route_survives_request_span_name_override` (span name is hard-coded via `schematize_url_operation`). ## Testing - 27 unit tests in `tests/appsec/appsec/api_security/test_normalized_route.py` covering named groups, positional groups, multi-param combining (`+`), `/?` stripping, explicit trailing-slash preservation, URL-encoding, mismatch detection, and `lru_cache` identity. - Integration tests in `test_tornado.py` exercise the full request path: named-group params (`/asm/{param_int}/{param_str}`), multi-param-in-segment (`/multi-param/{first+last}`), catch-all (`/files/{file_path}`), root (`/`), ASM-disabled gate, and API-Security-disabled gate. - `test_normalized_route_disabled_when_api_security_off` verifies the tag is absent when API Security is off while ASM is on. ## Risks None — all production code is confined to `ddtrace/appsec/`. No changes to integration code or public API. The `/asm/` route change (positional → named groups) is backward-compatible: Tornado passes named groups as keyword arguments that match the existing handler signature. ## Additional Notes Tornado routes using positional groups (no `(?P<name>...)`) produce auto-numbered placeholders (`{param1}`, `{param2}`, …). Users who want named parameters in `_dd.appsec.normalized_route` should use named capturing groups in their route patterns. Co-authored-by: christophe.papazian <[email protected]>
APPSEC-65477
Description
Extends RFC-1103
_dd.appsec.normalized_routeto Flask, following the FastAPI/Starlette (#17920) and Django (#18209) implementations.ddtrace/appsec/_api_security/_normalized_route.py):normalize_route_flask()with fast path (single-converter params) and slow path (multi-param segments,<path:name>catch-all,any()converters, static URL-encoding).ddtrace/appsec/_handlers.py): adds"flask"to_NORMALIZED_ROUTE_BY_INTEGRATION; resolvesDispatcherMiddlewaremount prefix fromFLASK_RESOURCE_FULLso sub-app routes include their full assembled path.Testing
normalize_route_flask(fast path, slow path, invalid inputs, DM assembly, edge cases).test_normalized_routeandtest_normalized_route_disabled_when_api_security_off.Risks
None — all production code is in
ddtrace/appsec/. No changes to integration code or public API.Additional Notes
Non-terminal
<path:...>routes (e.g./<path:wiki>/edit, valid in Werkzeug) returnNone— omit rather than guess, per RFC-1103 rule 5 which mandates catch-alls be terminal.