Skip to content

chore(ddtrace/tracer): fix flaky TestHealthMetricsRaceCondition#3971

Merged
dd-mergequeue[bot] merged 2 commits into
mainfrom
felix.geisendoerfer/fix-flaky-TestHealthMetricsRaceCondition
Sep 16, 2025
Merged

chore(ddtrace/tracer): fix flaky TestHealthMetricsRaceCondition#3971
dd-mergequeue[bot] merged 2 commits into
mainfrom
felix.geisendoerfer/fix-flaky-TestHealthMetricsRaceCondition

Conversation

@felixge

@felixge felixge commented Sep 16, 2025

Copy link
Copy Markdown
Member

What does this PR do?

Switch wg.Wait() and flush(5), otherwise we might flush before the wait group is done.

Stop relying on the total number of reported statsd metrics as an indicator for the statsd client having received the metrics we are looking for. In my testing, I've seen the client receive 200+ metrics in this test (should it?), so performing our check after only 10 metrics have reported seemed brittle.

Once we are on go1.25, we should start using the synctest package for this test. The changes in this patch are expected to fix the flakiness in a practical sense. But scheduling delays of > 5s will still cause this test to flake.

[1] https://tip.golang.org/doc/go1.25#new-testingsynctest-package

Motivation

This test keeps flaking. See this link.

2025-09-16 CI Test Runs  Datadog at 10 33 45@2x

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added.
  • New code is free of linting errors. You can check this by running ./scripts/lint.sh locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

Switch wg.Wait() and flush(5), otherwise we might flush before the wait
group is done.

Stop relying on the total number of reported statsd metrics as an
indicator for the statsd client having received the metrics we are
looking for. In my testing, I've seen the client receive 200+ metrics in
this test (should it?), so performing our check after only 10 metrics
have reported seemed brittle.

Once we are on go1.25, we should start using the synctest package for
this test. The changes in this patch are expected to fix the flakiness
in a practical sense. But scheduling delays of > 5s will still cause
this test to flake.

[1] https://tip.golang.org/doc/go1.25#new-testingsynctest-package
@felixge
felixge requested a review from a team as a code owner September 16, 2025 08:31
@pr-commenter

pr-commenter Bot commented Sep 16, 2025

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2025-09-16 18:00:42

Comparing candidate commit 1ff7d1c in PR branch felix.geisendoerfer/fix-flaky-TestHealthMetricsRaceCondition with baseline commit ce9b743 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 metrics, 0 unstable metrics.

@darccio darccio left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @felixge!

@felixge

felixge commented Sep 16, 2025

Copy link
Copy Markdown
Member Author

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Sep 16, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-09-16 17:50:04 UTC ℹ️ Start processing command /merge


2025-09-16 17:50:24 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2025-09-16 18:03:53 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 15m (p90).


2025-09-16 18:20:02 UTC ℹ️ MergeQueue: This merge request was merged

@dd-mergequeue
dd-mergequeue Bot merged commit a42a2ba into main Sep 16, 2025
244 checks passed
@dd-mergequeue
dd-mergequeue Bot deleted the felix.geisendoerfer/fix-flaky-TestHealthMetricsRaceCondition branch September 16, 2025 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants