Repository navigation
[bug] a slow runner turns main red: the SQLite job's 10-minute budget has 1.4x headroom over its p90 #594
Description
What happened
main went red at a981f935: the Test (SQLite backend) job was cancelled after 616 s against its timeout-minutes: 10 budget, and CI Green failed on test-sqlite='cancelled' (run 35092942280). The job had reached 74 % of the suite when the runner killed it. Re-running the failed jobs turned main green again with no code change, which is the whole point: the tree was fine, the budget was not.
The same job on the same tree took 266 s as the pull-request run of 33dc284a, the commit that a981f935 squashed. So one runner took at least 2.3× another for identical work.
Measured over the last 29 CI runs (2026-09-16): 27 successes between 234 s and 416 s, median 321 s, p90 343 s; the two non-successes are 47 s and 150 s cancellations from superseded pushes, not timeouts. The slowest success therefore uses 69 % of the 600 s cap, and the p90 uses 57 %. A runner 1.5× slower than the p90 exceeds the budget, which is what happened.
This is the second occurrence in a day: the same job was cancelled at 10 m 10 s on PR #584 and passed on a rerun.
Expected behavior
A slow runner does not turn main red. The budget still catches a genuinely hung suite rather than letting it run to the GitHub-imposed 6 h.
Evidence required to close
The new value justified by the measured distribution above, recorded where the current 10 comes from (ADR-0802), and a note of the p90 so that suite growth eating the new headroom stays visible.
Activity
ci(test-sqlite): set the job budget from its measured p90 (#594)
cdeust commentedon Sep 16, 2026
Correction to the numbers in this issue's description, caught by the review of PR #595.
I listed the last 30 CI runs regardless of date and read only each run's latest attempt. That sample crossed several days and hid every first attempt that was later rerun, which is where "29 runs, 27 successes, median 321 s, p90 343 s" and a supposed "150 s cancellation from a superseded push" came from. No such cancellation exists.
Re-measured attempt by attempt, scoped to 2026-09-16: 21 job executions, 18 successes from 234 s to 416 s, median 328 s, p90 348 s, and three non-successes. Two are real timeouts, not contention noise:
mainata981f935cancelled at 616 s and passing on its rerun in 416 s, and PR #584's first attempt cancelled at 610 s and passing on its rerun in 276 s. The third is a 47 s cancellation from a superseded push.The conclusion is unchanged and better supported: the former budget left 1.7x over the p90, and two executions in 21 exceeded it in a single day.
ci(test-sqlite): set the job budget from its measured p90 (#594)
ci(test-sqlite): set the job budget from its measured p90 (#594) (#595)