Add trace filter tests [APMSP-2763] by Eldolfin · Pull Request #6952 · DataDog/system-tests

Eldolfin · 2026-05-15T17:26:44Z

APMSP-2763
APMSP-3048

Motivation

Changes

Add system-tests coverage for the CSS V1.2 spec trace filters (filter_tags, filter_tags_regex, ignore_resources).

TODO

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

APMSP-2763

github-actions · 2026-05-15T17:27:16Z

CODEOWNERS have been resolved as:

tests/parametric/test_trace_filters.py                                  @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/parametric/test_library_tracestats.py                             @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

… scheme (#2008) These fields were renamed in version 7.79.0 of the agent for consistency. # What does this PR do? Allow both PascalCase and the newer snake_case format for these fields. # Motivation The trace exporter was crashing in a confusing way in new system tests I wrote for another feature: DataDog/system-tests#6952. I had an older agent. # Additional Notes Anything else we should know when reviewing? # How to test the change? Describe here in detail how the change can be validated. Co-authored-by: oscar.ledauphin <[email protected]>

# What does this PR do? Implements the CSS trace-level filtering mechanism, that is applied before stats computation. # Motivation It's un unmet requirement from the spec. # Additional Notes ~Please Ctrl+F `FIXME` when reviewing, I let some questions in the code that I'd like answers for before merging~ ## TODO: - [x] unit tests - [x] system-tests DataDog/system-tests#6952 - [x] normalization before filtering - [x] regex keys (currently only regex values are checked) - [x] new spec or CSS spec RFC - [x] telemetry ? Count dropped traces with the reason - [x] trim spaces when parsing config - [x] Move implementation to trace-utils - [x] stop using v4 span but generic spans instead for php - [x] move arcswap to the trace exporter (remove update_conf) - [x] use test agent for snapshot tests ## Wont do - [ ] ~advanced resource normalization (sql, redis etc...).~ No other implementation does that - [ ] ~100% normalization edge cases (empty service field + non-normalized name + filter depending on that)~ Too specific edge case. Ignoring it for now - [ ] ~check on meta_struct too~ Not in the spec [APMSP-2763] [APMSP-2763]: https://datadoghq.atlassian.net/browse/APMSP-2763?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: oscar.ledauphin <[email protected]>

datadog-official · 2026-06-17T15:00:23Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 85ab44e | Docs | Datadog PR Page | Give us feedback!}

…ter-test

chatgpt-codex-connector · 2026-07-02T13:00:14Z

💡 Codex Review

system-tests/tests/stats/test_trace_filters.py

Line 50 in 2e3ce18

    
           assert interfaces.library.wait_for(wait_function, 30), f"Timed out waiting for request with status {status_code}"

Avoid aborting scenarios from setup waits

This helper is called from the new setup_* methods, and in this framework pytest_collection_finish invokes those setup methods before tests run and closes targets/re-raises on any setup exception (conftest.py lines 449-478). If a tracer fails to emit the control stats payload within 30s, this assertion aborts the entire scenario during setup instead of reporting only the affected test failure; the wait result should be recorded and asserted in the matching test_* method.

system-tests/tests/test_the_test/scenarios.json

Lines 5641 to 5643 in 2e3ce18

    
           "tests/stats/test_trace_filters.py::Test_Trace_Filters_Regex_Require::test_filter_tags_regex_require": [ 
        
             "TRACE_STATS_COMPUTATION_TRACE_FILTER_REGEX_REQUIRE" 
        
           ],

Add the missing trace-filter scenarios to the map

compute_scenarios_by_files builds the scenarios for a modified test file solely from the nodeids present in this JSON map (utils/scripts/compute_libraries_and_scenarios.py lines 200-207). The added entries stop after the regex-require class, so changes to tests/stats/test_trace_filters.py will not select the three new edge-case scenarios (TRACE_STATS_COMPUTATION_TRACE_FILTER_REJECT_EDGE_CASES, TRACE_STATS_COMPUTATION_TRACE_FILTER_KEY_ONLY_REJECT, and TRACE_STATS_COMPUTATION_TRACE_FILTER_REQUIRE_EDGE_CASES) even though their tests were added in the same file.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ter-test

…, manifests

VianneyRuhlmann · 2026-07-06T11:17:10Z

+        """A require filter drops traces whose root span lacks the required tag."""
+        with test_library, test_library.dd_start_span(name="web.request"):
+            pass
+        assert not test_agent.traces(), "trace lacking the required tag should be dropped"


nit: why not check for stats here ?

These tests are focused on trace filters specifically, so they should not fail if CSS has a problem.

When I tried checking stats here using php it failed for the same reason as why CSS tests are disabled for php:

missing_feature (CSS v1.2.0: dd-trace-php ships CSS since v1.19.0, but the sidecar stats exporter retains buckets younger than ~20s (buffer_len=2 * bucket_size=10s) and dd_trace_synchronous_flush passes force=false, so short-lived parametric tests never observe a /v0.6/stats request)

cbeauchesne

From framework usage, all good.

I let someone familiar with the feature validate the test logic and approve the PR

ichinaski · 2026-07-07T09:49:06Z

+) -> pytest.MarkDecorator:
+    """Configure agent-side trace filters and pin the test agent version.
+
+    Filters are advertised in the test agent's /info response (via DD_AGENT_EXTRA_INFO) so the


Why is this behind an env var? The real agent always exposes them (since 7.75)

I'm not sure I understand what you're asking.
To clarify the env var DD_AGENT_EXTRA_INFO is used to add the trace filter configuration to the test agent /info response, AFAIK there is no other way to configure trace filters in the test agent.

And with the real agent we could've used the official parameters

ichinaski · 2026-07-07T10:02:00Z

  tests/parametric/test_telemetry.py::Test_TelemetrySCAEnvVar: v2.50.0
  tests/parametric/test_telemetry.py::Test_TelemetrySCAEnvVar::test_telemetry_sca_enabled_propagated_specifics: irrelevant
  tests/parametric/test_telemetry.py::Test_TelemetrySSIConfigs: v2.53.0
+  tests/parametric/test_trace_filters.py::Test_Trace_Filters: missing_feature


I believe dotnet does have support for this feature? https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6378587288/Implementation+Status#6.-Trace-Filters-(root-span%2C-applied-BEFORE-stats)

cc/ @andrewlock could you confirm whether this should be marked as implemented?

Looking at the test results, I see that 2 tests are failing for dotnet:
test_trace_filters_ignore_resources -> surprised this fails, it's a basic case
test_trace_filters_trim -> edge case behavior where the tag filter reject_tag : true is trimmed to be equivalent to reject_tag:true

mabdinur · 2026-07-13T14:49:35Z

+    tracer applies them client-side. The agent version is pinned to the minimum that supports
+    client-side stats according to the spec.
+    """
+    agent_version = MIN_AGENT_VERSION_FOR_CSS


Should we hard code the agent version here? It would be nice to have at least one test case that runs against the latest version of the agent

I'm not sure how I would go about it tbh.
I can't just let the default version of the test agent (test won't work because it's not in the \d+.\d+.\d+ format)

This only seems possible by using the real agent, so with end-to-end tests (or by using a quick hack where I fetch it from the github release page...)

These tests were previously end-to-end so if you want I can partially revert my deletion of end-to-end tests ?

cbeauchesne · 2026-07-17T09:00:34Z

+import pytest
+
+from .conftest import APMLibrary
+from .test_library_tracestats import MIN_AGENT_VERSION_FOR_CSS, enable_tracestats


Can you move MIN_AGENT_VERSION_FOR_CSS and enable_tracestats into tests/parametric/utils.py ? (long story short, because of the CI orchestrator that determine which scneario to run, a test file should never import something from another test file.)

feat: add trace filter tests (checked on stats)

633a7a5

APMSP-2763

Eldolfin changed the title ~~feat: add trace filter tests (checked on stats)~~ feat: add trace filter tests [APMSP-2763] May 15, 2026

Eldolfin changed the title ~~feat: add trace filter tests [APMSP-2763]~~ Add trace filter tests [APMSP-2763] May 15, 2026

This was referenced May 15, 2026

CSS v1.2.0: Phase A — parametric coverage additions #6938

Merged

feat(data-pipeline)!: CSS Trace Filters DataDog/libdatadog#1985

Merged

fix: simplify tests and explain results

6c4631b

Eldolfin mentioned this pull request May 18, 2026

fix(data-pipeline): allow old PascalCase fields in obfuscation config scheme DataDog/libdatadog#2008

Merged

add manifest

c358546

Eldolfin added 11 commits June 17, 2026 17:15

Merge remote-tracking branch 'origin/main' into oscarld/add-trace-fil…

7173a8e

…ter-test

fix: bad merge

3f0500b

test: edge cases

07511e1

feat: enable for php >=1.22.0

ca48ec8

fix: format

d0de68d

Merge remote-tracking branch 'origin/main' into oscarld/add-trace-fil…

d1e0ac2

…ter-test

fix: add new scenarios to ci

23908c8

fix: stop filtering on http.status_code

37b940b

Merge remote-tracking branch 'origin/main' into oscarld/add-trace-fil…

6db12ce

…ter-test

fix(manifest): add missing_feature to cpp_httpd/nginx

a93ba6a

Merge remote-tracking branch 'origin/main' into oscarld/add-trace-fil…

2e3ce18

…ter-test

Eldolfin marked this pull request as ready for review July 2, 2026 12:54

Eldolfin requested review from a team as code owners July 2, 2026 12:54

Eldolfin requested review from P403n1x87, brettlangdon, dromanol, dubloom and manuel-alvarez-alvarez and removed request for a team July 2, 2026 12:54

cbeauchesne reviewed Jul 2, 2026

View reviewed changes

Comment thread utils/_context/_scenarios/__init__.py Outdated

Eldolfin added 3 commits July 2, 2026 16:28

fix: add missing scenarios in scenarios.json

941cdf9

wip change to parametric

a9a1b37

Merge remote-tracking branch 'origin/main' into oscarld/add-trace-fil…

4ca7cac

…ter-test

Eldolfin requested a review from a team as a code owner July 3, 2026 18:37

Eldolfin requested review from mabdinur and removed request for a team July 3, 2026 18:37

Eldolfin added 2 commits July 3, 2026 20:37

revert: delete end-to-end filter tests changes

a380c14

better comments & assertion message, negative cases, fix env tag test…

1a7794c

…, manifests

Eldolfin force-pushed the oscarld/add-trace-filter-test branch from 97052c0 to 1a7794c Compare July 3, 2026 18:38

Eldolfin requested review from VianneyRuhlmann and ichinaski July 6, 2026 11:08

VianneyRuhlmann approved these changes Jul 6, 2026

View reviewed changes

cbeauchesne reviewed Jul 6, 2026

View reviewed changes

Eldolfin added 3 commits July 6, 2026 14:27

fix: add missing test_trace_filter_reject_missing tests

eb8a841

fix(manifest): remove useless comments

d5ef4d0

fix: fmt

85ab44e

ichinaski reviewed Jul 7, 2026

View reviewed changes

mabdinur approved these changes Jul 13, 2026

View reviewed changes

cbeauchesne requested changes Jul 17, 2026

View reviewed changes

Uh oh!

Conversation

Eldolfin commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

TODO

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Jun 17, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Jul 2, 2026

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eldolfin Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Eldolfin commented May 15, 2026 •

edited

Loading

github-actions Bot commented May 15, 2026 •

edited

Loading

datadog-official Bot commented Jun 17, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

Eldolfin Jul 7, 2026 •

edited

Loading