fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it by machine424 · Pull Request #17490 · prometheus/prometheus

machine424 · 2025-11-06T18:14:46Z

Improve test stability by waiting for the relevant metrics to appear on /metrics before the
first check on the desired shard count.

Increase the scrape interval to avoid timeouts, as 100 ms may be insufficient for Prometheus
to scrape itself in some environments (e.g., CI).

Have Prometheus scrape itself multiple times to increase the volume of data sent and help
fill the queue more quickly.

Which issue(s) does the PR fix:

Fixes #17489

Does this PR introduce a user-facing change?

NONE

machine424 · 2025-11-17T08:02:14Z

Of course, after tweaking, the test still detect the bug in v3.7.0 (the test still fails there)

$ go test -timeout 30000s -run ^TestRemoteWrite_ReshardingWithoutDeadlock$ ./cmd/prometheus -race --count=1 -v
=== RUN   TestRemoteWrite_ReshardingWithoutDeadlock
=== PAUSE TestRemoteWrite_ReshardingWithoutDeadlock
=== CONT  TestRemoteWrite_ReshardingWithoutDeadlock
    reload_test.go:194: time=2025-11-17T08:57:42.770+01:00 level=INFO source=main.go:1549 msg="updated GOGC" old=100 new=75
    reload_test.go:194: time=2025-11-17T08:57:42.771+01:00 level=INFO source=main.go:680 msg="Leaving GOMAXPROCS=12: CPU quota undefined" component=automaxprocs
    reload_test.go:194: time=2025-11-17T08:57:42.771+01:00 level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=30923764531 previous=9223372036854775807
    reload_test.go:194: time=2025-11-17T08:57:42.771+01:00 level=INFO source=main.go:722 msg="No time or size retention was set so using the default time retention" duration=15d
    reload_test.go:194: time=2025-11-17T08:57:42.771+01:00 level=INFO source=main.go:773 msg="Starting Prometheus Server" mode=server version="(version=, branch=, revision=unknown)"
    reload_test.go:194: time=2025-11-17T08:57:42.771+01:00 level=INFO source=main.go:778 msg="operational information" build_context="(go=go1.25.0, platform=darwin/arm64, user=, date=, tags=unknown)" host_details=(darwin) fd_limits="(soft=122880, hard=unlimited)" vm_limits="(soft=unlimited, hard=unlimited)"
    reload_test.go:194: time=2025-11-17T08:57:42.776+01:00 level=INFO source=web.go:660 msg="Start listening for connections" component=web address=0.0.0.0:54207
    reload_test.go:194: time=2025-11-17T08:57:42.776+01:00 level=INFO source=main.go:1293 msg="Starting TSDB ..."
    reload_test.go:194: time=2025-11-17T08:57:42.780+01:00 level=INFO source=tls_config.go:346 msg="Listening on" component=web address=[::]:54207
    reload_test.go:194: time=2025-11-17T08:57:42.781+01:00 level=INFO source=tls_config.go:349 msg="TLS is disabled." component=web http2=false address=[::]:54207
    reload_test.go:194: time=2025-11-17T08:57:42.783+01:00 level=INFO source=head.go:669 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
    reload_test.go:194: time=2025-11-17T08:57:42.783+01:00 level=INFO source=head.go:755 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=28.125µs
    reload_test.go:194: time=2025-11-17T08:57:42.783+01:00 level=INFO source=head.go:763 msg="Replaying WAL, this may take a while" component=tsdb
    reload_test.go:194: time=2025-11-17T08:57:42.784+01:00 level=INFO source=head.go:836 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=489.75µs
    reload_test.go:194: time=2025-11-17T08:57:42.784+01:00 level=INFO source=head.go:873 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=183.917µs wal_replay_duration=529.458µs wbl_replay_duration=125ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=28.125µs total_replay_duration=790.334µs
    reload_test.go:194: time=2025-11-17T08:57:42.793+01:00 level=INFO source=main.go:1314 msg="filesystem information" fs_type=1a
    reload_test.go:194: time=2025-11-17T08:57:42.794+01:00 level=INFO source=main.go:1317 msg="TSDB started"
    reload_test.go:194: time=2025-11-17T08:57:42.794+01:00 level=DEBUG source=main.go:1318 msg="TSDB options" MinBlockDuration=2h MaxBlockDuration=1d12h MaxBytes=0B NoLockfile=false RetentionDuration=15d WALSegmentSize=0B WALCompressionType=snappy
    reload_test.go:194: time=2025-11-17T08:57:42.794+01:00 level=INFO source=main.go:1502 msg="Loading configuration file" filename=/var/folders/pl/4zxvwhbd0q5bkw9vxnk08swm0000gn/T/TestRemoteWrite_ReshardingWithoutDeadlock3889829689/001/prometheus.yml
    reload_test.go:194: time=2025-11-17T08:57:42.795+01:00 level=INFO source=watcher.go:240 msg="Starting WAL watcher" component=remote remote_name=fce6dd url=http://127.0.0.1:54208 queue=fce6dd
    reload_test.go:194: time=2025-11-17T08:57:42.795+01:00 level=INFO source=metadata_watcher.go:90 msg="Starting scraped metadata watcher" component=remote remote_name=fce6dd url=http://127.0.0.1:54208
    reload_test.go:194: time=2025-11-17T08:57:42.795+01:00 level=INFO source=watcher.go:292 msg="Replaying WAL" component=remote remote_name=fce6dd url=http://127.0.0.1:54208 queue=fce6dd
    reload_test.go:194: time=2025-11-17T08:57:42.795+01:00 level=DEBUG source=watcher.go:312 msg="Tailing WAL" component=remote remote_name=fce6dd url=http://127.0.0.1:54208 lastCheckpoint="" checkpointIndex=0 currentSegment=0 lastSegment=0
    reload_test.go:194: time=2025-11-17T08:57:42.795+01:00 level=DEBUG source=watcher.go:318 msg="Processing segment" component=remote remote_name=fce6dd url=http://127.0.0.1:54208 currentSegment=0
    reload_test.go:194: time=2025-11-17T08:57:42.812+01:00 level=DEBUG source=manager.go:305 msg="Starting provider" component="discovery manager scrape" provider=static/0 subs=map[self:{}]
    reload_test.go:194: time=2025-11-17T08:57:42.812+01:00 level=DEBUG source=manager.go:345 msg="Discoverer channel closed" component="discovery manager scrape" provider=static/0
    reload_test.go:194: time=2025-11-17T08:57:42.812+01:00 level=INFO source=main.go:1542 msg="Completed loading of configuration file" db_storage=2.167µs remote_storage=1.121166ms web_handler=1.709µs query_engine=2.375µs scrape=16.292417ms scrape_sd=124.917µs notify=3.25µs notify_sd=2.75µs rules=2.792µs tracing=31.333µs filename=/var/folders/pl/4zxvwhbd0q5bkw9vxnk08swm0000gn/T/TestRemoteWrite_ReshardingWithoutDeadlock3889829689/001/prometheus.yml totalDuration=18.222333ms
    reload_test.go:194: time=2025-11-17T08:57:42.812+01:00 level=INFO source=main.go:1278 msg="Server is ready to receive web requests."
    reload_test.go:194: time=2025-11-17T08:57:42.812+01:00 level=INFO source=manager.go:190 msg="Starting rule manager..." component="rule manager"
    reload_test.go:194: time=2025-11-17T08:57:48.268+01:00 level=INFO source=watcher.go:538 msg="Done replaying WAL" component=remote remote_name=fce6dd url=http://127.0.0.1:54208 duration=5.472887458s
    reload_test.go:194: time=2025-11-17T08:57:52.796+01:00 level=DEBUG source=queue_manager.go:1156 msg=QueueManager.calculateDesiredShards component=remote remote_name=fce6dd url=http://127.0.0.1:54208 dataInRate=0.6 dataOutRate=0.4 dataPendingRate=0.19999999999999996 dataPending=0 dataOutDuration=0.40072735010000005 timePerSample=1.00181837525 desiredShards=0.60109102515 highestSent=1.763366268e+09 highestRecv=1.763366268e+09
    reload_test.go:194: time=2025-11-17T08:57:52.796+01:00 level=DEBUG source=queue_manager.go:1173 msg=QueueManager.updateShardsLoop component=remote remote_name=fce6dd url=http://127.0.0.1:54208 lowerBound=0.7 desiredShards=0.60109102515 upperBound=1.3
    main_test.go:1045: 
        	Error Trace:	/xxx//prometheus/cmd/prometheus/main_test.go:1045
        	Error:      	Condition never satisfied
        	Test:       	TestRemoteWrite_ReshardingWithoutDeadlock
--- FAIL: TestRemoteWrite_ReshardingWithoutDeadlock (30.87s)
FAIL
FAIL	github.com/prometheus/prometheus/cmd/prometheus	32.532s
FAIL

…le and re-enable it Improve test stability by waiting for the relevant metrics to appear on /metrics before the first check on the desired shard count. Increase the scrape interval to avoid timeouts, as 100 ms may be insufficient for Prometheus to scrape itself in some environments (e.g., CI). Have Prometheus scrape itself multiple times to increase the volume of data sent and help fill the queue more quickly. Signed-off-by: machine424 <[email protected]>

bwplotka

Amazing, thanks!

machine424 requested review from bwplotka, cstyan and tomwilkie as code owners November 6, 2025 18:14

bwplotka mentioned this pull request Nov 12, 2025

Flaky test TestRemoteWrite_ReshardingWithoutDeadlock #17489

Closed

This was referenced Nov 13, 2025

WIP Debug flaky test #17511

Closed

tests: make TestRemoteWrite_ReshardingWithoutDeadlock less flaky #17522

Closed

machine424 force-pushed the rrtt branch 2 times, most recently from 708abcb to 4bf97f6 Compare November 16, 2025 18:33

machine424 changed the title ~~WIP: debug TestRemoteWrite_ReshardingWithoutDeadlock in CI~~ fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it Nov 16, 2025

machine424 changed the title ~~fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it~~ WIP: fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it Nov 16, 2025

machine424 force-pushed the rrtt branch 2 times, most recently from e776a97 to 2c17cd1 Compare November 17, 2025 08:01

machine424 force-pushed the rrtt branch from 2c17cd1 to 99eb148 Compare November 17, 2025 08:19

machine424 changed the title ~~WIP: fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it~~ fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it Nov 17, 2025

machine424 force-pushed the rrtt branch from 99eb148 to 49c7416 Compare November 17, 2025 08:22

bwplotka approved these changes Nov 25, 2025

View reviewed changes

bwplotka merged commit e799952 into prometheus:main Nov 25, 2025
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it#17490

fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it#17490
bwplotka merged 1 commit intoprometheus:mainfrom
machine424:rrtt

machine424 commented Nov 6, 2025 •

edited

Loading

Uh oh!

machine424 commented Nov 17, 2025 •

edited

Loading

Uh oh!

bwplotka left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

machine424 commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

machine424 commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

machine424 commented Nov 6, 2025 •

edited

Loading

machine424 commented Nov 17, 2025 •

edited

Loading