Skip to content

chore: enable shuffle testing for most core packages#4061

Merged
dd-mergequeue[bot] merged 3 commits into
mainfrom
felix.geisendoerfer/chore-enable-shuffle-testing-for-most-core-packages
Oct 23, 2025
Merged

chore: enable shuffle testing for most core packages#4061
dd-mergequeue[bot] merged 3 commits into
mainfrom
felix.geisendoerfer/chore-enable-shuffle-testing-for-most-core-packages

Conversation

@felixge

@felixge felixge commented Oct 22, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

Enable test shuffling for most of our core packages.

Packages that were unable to successfully pass their tests with -shuffle on for 10 times in a row (on my local machine) have been excluded. See #4060 for details.

Additionally the following packages were excluded because they flaked in CI (this may or may not be due to shuffle):

The plan is to enable shuffling for the excluded packages in the future once their test suite has been fixed. Enabling shuffle testing for contrib packages will also be done separately.

If you see test flakes, and suspect that they are caused by this PR, please try to fix them instead of adding the packages to the shuffle exclusion list.

PROF-12813

Motivation

Some of our tests have come to rely on global state pollution over time. I.e. they need another test to run first in order to pass, or rely on another test not running first in order to pass. This is undesirable as it often causes problems for people debugging test failures (e.g. test passes when go test -run TestXXX but not when running all tests in the pkg). It may also hide implementation bugs.

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added.
  • New code is free of linting errors. You can check this by running ./scripts/lint.sh locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

@pr-commenter

pr-commenter Bot commented Oct 22, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2025-10-22 16:11:33

Comparing candidate commit 685c422 in PR branch felix.geisendoerfer/chore-enable-shuffle-testing-for-most-core-packages with baseline commit be655e9 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 metrics, 0 unstable metrics.

@felixge
felixge force-pushed the felix.geisendoerfer/chore-enable-shuffle-testing-for-most-core-packages branch from 4e8aace to 74f17d8 Compare October 22, 2025 14:23
@datadog-datadog-prod-us1

This comment has been minimized.

Sign in to view
@felixge
chore: add instrumentation/appsec pkg to shuffle exclude list
3265355 
Can't reproduce it locally, but flaked in CI as well.
@felixge
felixge force-pushed the felix.geisendoerfer/chore-enable-shuffle-testing-for-most-core-packages branch from 55b0546 to 3265355 Compare October 22, 2025 15:30
@felixge
felixge marked this pull request as ready for review October 22, 2025 19:05
@felixge
felixge requested a review from a team as a code owner October 22, 2025 19:05
nsrip-dd
nsrip-dd reviewed Oct 22, 2025
View reviewed changes
Comment thread scripts/ci_test_core.sh
nsrip-dd
nsrip-dd approved these changes Oct 22, 2025
View reviewed changes

@nsrip-dd nsrip-dd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed that the tests are split up appropriately in the CI run: https://github.com/DataDog/dd-trace-go/actions/runs/18722222366/job/53397550605?pr=4061#step:6:237

I also confirmed locally that gotestsum will log the shuffle seed if the tests fail:

% gotestsum -- -v -shuffle=on ./profiler
✖  profiler (21.383s) (-test.shuffle 1761161421624258000)

=== Skipped
=== SKIP: profiler TestDebugCompressionEnv (0.00s)
    compression_test.go:77: Flaky. See #3681

=== Failed
=== FAIL: profiler TestFail (0.00s)
    profiler_test.go:948: fail!

DONE 152 tests, 1 skipped, 1 failure in 21.383s

Thanks for doing this!

@felixge

felixge commented Oct 23, 2025

Copy link
Copy Markdown
Member Author

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Oct 23, 2025 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-10-23 04:05:07 UTC ℹ️ Start processing command /merge

2025-10-23 04:05:12 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 18m (p90).

2025-10-23 04:20:04 UTC ℹ️ MergeQueue: This merge request was merged

@dd-mergequeue
dd-mergequeue Bot merged commit 56a1413 into main Oct 23, 2025
250 checks passed
@dd-mergequeue
dd-mergequeue Bot deleted the felix.geisendoerfer/chore-enable-shuffle-testing-for-most-core-packages branch October 23, 2025 04:19
@felixge felixge mentioned this pull request Oct 23, 2025
Draft
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nsrip-dd nsrip-dd nsrip-dd approved these changes

Assignees

No one assigned

Labels

mergequeue-status: done

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@felixge @nsrip-dd