Skip to content

tsdb: fix flaky TestBlockRanges by using explicit compaction#18085

Merged
bwplotka merged 1 commit into
prometheus:mainfrom
aknuds1:arve/fix-test-block-ranges
Feb 17, 2026
Merged

tsdb: fix flaky TestBlockRanges by using explicit compaction#18085
bwplotka merged 1 commit into
prometheus:mainfrom
aknuds1:arve/fix-test-block-ranges

Conversation

@aknuds1

@aknuds1 aknuds1 commented Feb 14, 2026

Copy link
Copy Markdown
Contributor

Which issue(s) does the PR fix:

N/A — fixes a flaky test discovered during local testing.

Does this PR introduce a user-facing change?

NONE

Summary

TestBlockRanges and TestBlockRanges_AppendV2 are flaky because they rely on background auto-compaction via the run() goroutine, using a polling loop with a 10-second effective timeout. On slow or loaded CI machines, goroutine scheduling delays can cause this timeout to be insufficient.

This PR fixes the flakiness by:

  • Replacing polling loops with explicit db.Compact(ctx) calls — makes compaction deterministic, removing all timing dependency
  • Calling db.DisableCompactions() after each Open() — prevents the background goroutine from racing with explicit compaction
  • Fixing the overlap assertion direction (LessOrEqualGreaterOrEqual) — the original assertion checked adjacency, not non-overlap, which is what the test comment describes
  • Storing db.Blocks() in a local variable before assertions — eliminates a potential TOCTOU race if background compaction were to run between calls

This approach is consistent with other tests in the codebase (e.g., TestNoEmptyBlocks, TestDeleteCompactionBlockAfterFailedReload) that already use explicit db.Compact() with db.DisableCompactions().

Verified by running both tests 100 times each with -race — all iterations passed.

@aknuds1
aknuds1 force-pushed the arve/fix-test-block-ranges branch from 25a2414 to 283cc4e Compare February 14, 2026 15:42
@aknuds1
aknuds1 force-pushed the arve/fix-test-block-ranges branch from 283cc4e to ef5ce89 Compare February 15, 2026 09:54
@aknuds1
aknuds1 marked this pull request as draft February 15, 2026 09:59
@aknuds1 aknuds1 changed the title tsdb: fix flaky TestBlockRanges by using explicit compaction WIP: tsdb: fix flaky TestBlockRanges by using explicit compaction Feb 15, 2026
Replace polling loops (for range 100 { time.Sleep }) with explicit
db.Compact() calls after disabling background compaction, eliminating
CI flakiness on slow machines. Also fix incorrect overlap assertions
that were checking the wrong direction (LessOrEqual -> GreaterOrEqual).

Signed-off-by: Arve Knudsen <[email protected]>
@aknuds1
aknuds1 force-pushed the arve/fix-test-block-ranges branch from 5a61c87 to b0718d5 Compare February 15, 2026 10:44
@aknuds1 aknuds1 changed the title WIP: tsdb: fix flaky TestBlockRanges by using explicit compaction tsdb: fix flaky TestBlockRanges by using explicit compaction Feb 15, 2026
@aknuds1
aknuds1 marked this pull request as ready for review February 15, 2026 11:09

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bwplotka
bwplotka merged commit e4c997c into prometheus:main Feb 17, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants