Skip to content

feat(asm): add normalized HTTP route span tag for Flask#18343

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits into
mainfrom
christophe-papazian/appsec-normalized-route-flask
Jun 1, 2026
Merged

feat(asm): add normalized HTTP route span tag for Flask#18343
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits into
mainfrom
christophe-papazian/appsec-normalized-route-flask

Conversation

@christophe-papazian

Copy link
Copy Markdown
Contributor

APPSEC-65477

Description

Extends RFC-1103 _dd.appsec.normalized_route to Flask, following the FastAPI/Starlette (#17920) and Django (#18209) implementations.

  • Normalizer (ddtrace/appsec/_api_security/_normalized_route.py): normalize_route_flask() with fast path (single-converter params) and slow path (multi-param segments, <path:name> catch-all, any() converters, static URL-encoding).
  • Handler (ddtrace/appsec/_handlers.py): adds "flask" to _NORMALIZED_ROUTE_BY_INTEGRATION; resolves DispatcherMiddleware mount prefix from FLASK_RESOURCE_FULL so sub-app routes include their full assembled path.

Testing

  • 56 new unit test cases for normalize_route_flask (fast path, slow path, invalid inputs, DM assembly, edge cases).
  • Integration tests un-xfailed for Flask on test_normalized_route and test_normalized_route_disabled_when_api_security_off.
  • Covers both flat-app (Blueprint) and subapp (DispatcherMiddleware) interface variants.

Risks

None — all production code is in ddtrace/appsec/. No changes to integration code or public API.

Additional Notes

Non-terminal <path:...> routes (e.g. /<path:wiki>/edit, valid in Werkzeug) return None — omit rather than guess, per RFC-1103 rule 5 which mandates catch-alls be terminal.

@christophe-papazian
christophe-papazian force-pushed the christophe-papazian/appsec-normalized-route-flask branch from 7330a2d to 4f8fa26 Compare May 29, 2026 15:50
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented May 29, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/appsec/_api_security/_normalized_route.py                       @DataDog/asm-python
ddtrace/appsec/_handlers.py                                             @DataDog/asm-python
releasenotes/notes/appsec-normalized-route-flask-d51f53c0fded450e.yaml  @DataDog/apm-python
tests/appsec/appsec/api_security/test_normalized_route.py               @DataDog/asm-python
tests/appsec/contrib_appsec/flask_app/app.py                            @DataDog/asm-python
tests/appsec/contrib_appsec/flask_app/app_subapps.py                    @DataDog/asm-python
tests/appsec/contrib_appsec/test_flask.py                               @DataDog/asm-python
tests/appsec/contrib_appsec/utils.py                                    @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_osspawn[flask_appsec_good_rules_env].json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_osspawn[flask_appsec_good_rules_env]_220.json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_ossystem[flask_appsec_good_rules_env].json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_ossystem[flask_appsec_good_rules_env]_220.json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_subprocesscommunicatenoshell[flask_appsec_good_rules_env].json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_subprocesscommunicatenoshell[flask_appsec_good_rules_env]_220.json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_subprocesscommunicateshell[flask_appsec_good_rules_env].json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_processexec_subprocesscommunicateshell[flask_appsec_good_rules_env]_220.json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_userblock_match_200_json[flask_appsec_good_rules_env].json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_userblock_match_200_json[flask_appsec_good_rules_env]_220.json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_userblock_match_403_json[flask_appsec_good_rules_env].json  @DataDog/asm-python
tests/snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_userblock_match_403_json[flask_appsec_good_rules_env]_220.json  @DataDog/asm-python

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 8 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]   View in Datadog   GitLab

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration. NotImplementedError: This version of CPython is not supported yet

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64, 1]   View in Datadog   GitLab

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration. NotImplementedError: This version of CPython is not supported yet

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]   View in Datadog   GitLab

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration. NotImplementedError: This version of CPython is not supported yet

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6e959aa | Docs | Datadog PR Page | Give us feedback!

@christophe-papazian christophe-papazian added the ASM Application Security Monitoring label May 29, 2026
@pr-commenter

pr-commenter Bot commented May 29, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-01 10:47:23

Comparing candidate commit 6e959aa in PR branch christophe-papazian/appsec-normalized-route-flask with baseline commit 86fc5da in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 83 metrics, 0 unstable metrics.

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+108.507µs; +117.853µs] or [+25.151%; +27.318%]

@christophe-papazian
christophe-papazian force-pushed the christophe-papazian/appsec-normalized-route-flask branch from 185e209 to 0e1ec61 Compare May 29, 2026 16:37
christophe-papazian and others added 4 commits June 1, 2026 11:01
Extends RFC-1103 ``_dd.appsec.normalized_route`` to Flask, mirroring the
FastAPI/Starlette (#17920) and Django (#18209) implementations.

- ``ddtrace/appsec/_api_security/_normalized_route.py``: adds
  ``normalize_route_flask`` with a fast path (single-converter params, no
  catch-all) and a slow path covering multi-param-in-segment, ``<path:name>``
  catch-all, ``any(...)`` converters, and static URL-encoding.
- ``ddtrace/appsec/_handlers.py``: adds ``"flask"`` to
  ``_NORMALIZED_ROUTE_BY_INTEGRATION``; resolves the DispatcherMiddleware
  mount prefix from ``flask.resource.full`` so sub-app routes include their
  full assembled path.
- Test app (flat + subapp): adds ``/multi-param/<first>.<last>/`` and
  ``/files/<path:file_path>`` routes.
- ``tests/appsec/contrib_appsec/utils.py``: removes Flask from
  ``xfail_interface`` on the two normalized-route tests.
- Unit tests: 50 new parametrized cases for ``normalize_route_flask``,
  fast-path eligibility, and invalid-input rejection.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Apply fixes from three rounds of dual review (Claude subagent + Codex):

_normalized_route.py
- _FLASK_PARAM_REGEX: change [^>:]+ to [^>]+ so converter arguments
  containing `:` (e.g. custom <regex("[a:b]+"): slug>) are handled via
  backtracking rather than stopping at the first `:`. Add clarifying comment.
- _normalize_route_flask_slow: document why non-terminal <path:...> segments
  return None — Werkzeug allows them at runtime but RFC-1103 rule 5 mandates
  catch-alls be terminal; omit-rather-than-guess is the safe contract.
- Add comment on catch-all-wins-segment behaviour for mixed path+non-path
  params (Werkzeug rejects these at registration, so unreachable in practice).

_handlers.py
- Import FLASK_RESOURCE_FULL from ddtrace.internal.constants instead of
  duplicating the string literal; removes the private _FLASK_RESOURCE_FULL_TAG
  alias so the constant is used directly at the call site.

test_normalized_route.py
- Move normalize_route_flask import to file top; remove noqa E402.
- Add assert normalize_route_flask(route) == expected to fast-path test so
  the public entry-point wiring is covered, not just the cached helper.
- Add test cases: param named <path> (no converter, not a catch-all),
  custom converter with : in argument string.
- Add test_normalize_route_flask_dm_assembly_via_handler: isolated unit test
  for the handler-level FLASK_RESOURCE_FULL assembly logic via mocks; use the
  imported FLASK_RESOURCE_FULL constant rather than a bare string literal.
- Document why /<path:tail>/and-after returns None (RFC-1103 design choice,
  not a Werkzeug constraint).

utils.py
- Add /multi-param/john.doe (no trailing slash) to test_normalized_route
  parametrize to cover the non-trailing-slash route registration.
- Document why Flask is kept in xfail for the request_span_name-override
  test (Flask middleware uses a fixed _request_call_name class attribute).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Three distinct CI failures on the PR:

1. Django/FastAPI appsec_threats tests: `test_normalized_route` was
   parametrized with `/multi-param/john.doe` (no trailing slash) added
   in a prior review cycle. Django and FastAPI only register the
   trailing-slash variant of that route, causing HTTP 301/308 redirects
   and assertion failures. Remove the no-trailing-slash case from the
   shared parametrize — the trailing-slash variant already exercises
   Flask multi-param normalization end-to-end, and unit tests in
   test_normalized_route.py cover both shapes.

2. Flask appsec_threats endpoint_discovery test: `endpoint_path_to_uri`
   only substituted `<int:name>` and `<str:name>` patterns. The new
   `/multi-param/<first>.<last>/` route uses bare `<name>` converters
   (no type prefix), causing the raw pattern to be used as the request
   URI, which returns 308 (redirect to trailing-slash variant). Add
   substitution rules for bare `<name>` and `<path:name>` catch-all,
   and add the two new routes to ENDPOINT_DISCOVERY_EXPECTED_PATHS.

3. Flask contrib snapshot tests: Flask now emits `_dd.appsec.normalized_route`
   for matched routes when API Security is active. The appsec snapshot
   tests (processexec_*, userblock_*) run with DD_APPSEC_ENABLED=true
   and reach matched routes, so the new tag now appears in received
   spans. Update all 12 affected snapshot files (both `_220` and legacy
   variants). The ipblock tests are not affected because they are
   blocked before route matching and do not emit the tag.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Two remaining failures after the previous fix:

1. Snapshot JSON trailing comma: `fmt-snapshots` introduced a trailing
   comma in the `meta` object of two `test_flask_processexec_osspawn`
   snapshot files, causing a `400: Illegal trailing comma before end of
   object` parse error in the test agent. Strip the trailing comma.

2. Endpoint discovery 308 redirect: registering both `/multi-param/<first>.<last>/`
   (trailing slash) and `/multi-param/<first>.<last>` (no trailing slash)
   causes the endpoint collection to discover both paths. `endpoint_path_to_uri`
   converts the bare `<name>` converters to `test`, yielding
   `/multi-param/test.test`, which Werkzeug redirects (308) to the
   trailing-slash variant — failing the 200/401 assertion.
   Remove the no-trailing-slash multi-param route from the Flask test apps
   (only the trailing-slash variant is needed for normalisation coverage).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
@christophe-papazian
christophe-papazian force-pushed the christophe-papazian/appsec-normalized-route-flask branch from 0e1ec61 to 0b5a52b Compare June 1, 2026 09:01
christophe-papazian and others added 2 commits June 1, 2026 11:12
The two osspawn snapshot files were formatted with Python's json.dump
(4-space indentation) instead of the test-agent canonical format
(2-space, compact). Run scripts/lint fmt-snapshots to restore the
correct format so the prechecks snapshot-format check passes.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
- Remove duplicate module-level DM comment from _handlers.py (covered by inline)
- Shorten multi-line comments in _handlers.py and _normalized_route.py to one-liners
- Add ("/files/<path>", "/files/{path}") to test_normalize_route_flask_fast_path
  to document that a param named "path" (no converter) takes the fast path,
  distinct from <path:name> which is the catch-all converter

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@christophe-papazian
christophe-papazian marked this pull request as ready for review June 1, 2026 12:08
@christophe-papazian
christophe-papazian requested review from a team as code owners June 1, 2026 12:09
@christophe-papazian

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 4b84f68 into main Jun 1, 2026
740 of 743 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the christophe-papazian/appsec-normalized-route-flask branch June 1, 2026 15:39
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jun 4, 2026
## Description

Extends RFC-1103 `_dd.appsec.normalized_route` to Tornado, following the FastAPI/Starlette (#17920), Django (#18209), and Flask (#18343) implementations.

Tornado's ddtrace integration produces `http.route` strings with `%s` as a placeholder for every capturing group (via `_regex_to_route`). The new normalizer maps those placeholders to parameter names sourced from `path_params`: a dict for named groups (`(?P<name>...)`), a list for positional groups, or `{}` for static routes. Optional trailing-slash patterns (`/?`) are treated as not declaring a trailing slash per RFC-1103 rule 1, consistent with the Django `^asm/?$` → `/asm` convention.

**Changes:**

- **`_normalized_route.py`** — adds `_normalize_route_tornado_cached` (LRU-cached, keyed on route + param-names tuple) and `normalize_route_tornado` (public entry point). Handles `%s` placeholder mapping, `/?` stripping, multi-param-in-segment combining with `+` (rule 5), and static-segment URL-encoding (rule 3).
- **`_handlers.py`** — registers `"tornado": normalize_route_tornado` in `_NORMALIZED_ROUTE_BY_INTEGRATION`.
- **`tornado_app/app.py`** — converts the `/asm/` route from positional to named capturing groups so `path_params` arrives as a dict with proper parameter names; adds `MultiParamHandler` (`/multi-param/`) and `FilesHandler` (`/files/`) for integration test coverage.
- **`test_tornado.py`** — overrides `test_normalized_route` and `test_normalized_route_disabled_when_api_security_off` with Tornado-specific expected values (no trailing slash for `/?` routes); expands `ENDPOINT_DISCOVERY_EXPECTED_PATHS`.
- **`test_normalized_route.py`** — 27 new unit tests for `normalize_route_tornado` covering all code paths.
- **`utils.py`** — documents why Tornado is skipped in `test_normalized_route_survives_request_span_name_override` (span name is hard-coded via `schematize_url_operation`).

## Testing

- 27 unit tests in `tests/appsec/appsec/api_security/test_normalized_route.py` covering named groups, positional groups, multi-param combining (`+`), `/?` stripping, explicit trailing-slash preservation, URL-encoding, mismatch detection, and `lru_cache` identity.
- Integration tests in `test_tornado.py` exercise the full request path: named-group params (`/asm/{param_int}/{param_str}`), multi-param-in-segment (`/multi-param/{first+last}`), catch-all (`/files/{file_path}`), root (`/`), ASM-disabled gate, and API-Security-disabled gate.
- `test_normalized_route_disabled_when_api_security_off` verifies the tag is absent when API Security is off while ASM is on.

## Risks

None — all production code is confined to `ddtrace/appsec/`. No changes to integration code or public API. The `/asm/` route change (positional → named groups) is backward-compatible: Tornado passes named groups as keyword arguments that match the existing handler signature.

## Additional Notes

Tornado routes using positional groups (no `(?P<name>...)`) produce auto-numbered placeholders (`{param1}`, `{param2}`, …). Users who want named parameters in `_dd.appsec.normalized_route` should use named capturing groups in their route patterns.

Co-authored-by: christophe.papazian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ASM Application Security Monitoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants