flaky: TestSandboxAssetPrewarmDownloads #1939

Flaky test detected

Test: TestSandboxAssetPrewarmDownloads
Job: unit (run-tests)

Evidence (last 7 days)

Fails intermittently across unrelated branches — cross-PR, so nondeterminism rather than any change under test. ~1.5% failure rate (≈6 failures / ≈400 runs in the rolling window), most recently on a benchmarking-only PR that does not touch cmd/atelet at all.

Failing runs:

• https://github.com/agent-substrate/substrate/actions/runs/35870777062 (main)
• https://github.com/agent-substrate/substrate/actions/runs/35898961636 (main)
• https://github.com/agent-substrate/substrate/actions/runs/35900685339 (remove-version-label)
• https://github.com/agent-substrate/substrate/actions/runs/36185288287 (metrics/rename-no-free-worker)
• https://github.com/agent-substrate/substrate/actions/runs/36255544447 (filecache-m2)
• https://github.com/agent-substrate/substrate/actions/runs/36444425006 (PR benchmarking: agent-session workload — a scripted coding agent that suspends while the LLM thinks #1934, benchmarking-only diff)

Root cause

It is a data race, and -race makes it fatal. The test patches nodepath.StaticFilesDir and prewarmMaxJitter and restores them in t.Cleanup, then returns as soon as the prewarmed asset appears on disk. The prewarm worker goroutine started by startSandboxAssetPrewarm is still running its tail at that point (the pause-image pull after the asset lands) and reads nodepath.StaticFilesDir in downloadVerified (cmd/atelet/sandbox_assets.go:285) while the cleanup writes it:

WARNING: DATA RACE
Write at 0x000006ca6210 by goroutine 891:
  cmd/atelet.TestSandboxAssetPrewarmDownloads.func1()  sandbox_prewarm_test.go:360  (t.Cleanup restoring the globals)
Previous read at 0x000006ca6210 by goroutine 937:
  cmd/atelet.(*AteomHerder).downloadVerified()         sandbox_assets.go:285
  cmd/atelet.(*sandboxPrewarmer).prewarm.func2()       sandbox_prewarm.go:270

Whether the race window is open depends on how slowly the worker's tail runs relative to the test's poll loop, which is why it mostly bites on loaded CI runners and passes locally.

Fix

The worker needs to be joinable: nothing today lets a caller wait for the prewarm goroutine to exit after context cancellation. A fix PR follows that makes startSandboxAssetPrewarm return the prewarmer with a wait() and has the test join the worker before restoring the globals.