|
| 1 | +<!-- Generated by Cortex wiki_reindex from wiki/adr/cortex/1074-the-sqlite-test-job-s-budget-is-set-from-its-measured-p90.md; ADR-1074. Read-only mirror: edit the canonical wiki page. --> |
| 2 | + |
| 3 | +--- |
| 4 | +created: 2026-09-16T15:45:52Z |
| 5 | +kind: adr |
| 6 | +number: 1074 |
| 7 | +status: accepted |
| 8 | +tags: [ci, workflows, timeout, issue-594] |
| 9 | +title: The SQLite test job's budget is set from its measured p90 |
| 10 | +--- |
| 11 | +# ADR-1074: The SQLite test job's budget is set from its measured p90 |
| 12 | + |
| 13 | +## Status |
| 14 | + |
| 15 | +accepted |
| 16 | + |
| 17 | +## Context |
| 18 | + |
| 19 | +`main` went red at `a981f935`: the `Test (SQLite backend)` job hit its `timeout-minutes: 10` budget after 616 s, having reached 74 % of the suite, and `CI Green` failed on `test-sqlite='cancelled'` (run 35092942280). Re-running the failed jobs turned `main` green with no code change. |
| 20 | + |
| 21 | +The same tree took 266 s as the pull-request run of `33dc284a`, the commit `a981f935` squashed, so one runner took at least 2.3× another for identical work. Measured over that day's 21 job executions, every attempt counted: 18 successes between 234 s and 416 s, median 328 s, p90 348 s, and three non-successes. Two of those three are timeouts, not contention noise: `main` at 616 s and PR #584's first attempt at 610 s, each passing on a rerun in 416 s and 276 s; the third is a 47 s cancellation from a superseded push. The slowest success used 69 % of the 600 s budget and the p90 used 58 %, so a runner three quarters slower than the p90 exceeds it, which happened twice in one day. |
| 22 | + |
| 23 | +The budget exists to catch a hung suite before GitHub's own 6-hour ceiling (ADR-0802), not to police runner speed. At 1.7× the p90 it was doing the second. |
| 24 | + |
| 25 | +The job runs the whole suite serially, `pytest --tb=short -q -p no:randomly`, on `ubuntu-latest`. Splitting it across workers would cut the wall clock, but the suite shares SQLite and PostgreSQL fixtures whose isolation under `pytest-xdist` is unproven here; that is a separate measurement, not a timeout decision. |
| 26 | + |
| 27 | +## Decision |
| 28 | + |
| 29 | +`Test (SQLite backend)` gets `timeout-minutes: 20`, which is 3.4× its measured p90, about twice the point where both timeouts fired, and still one eighteenth of GitHub's ceiling, and the workflow carries the measurement next to the number so that suite growth eating the new headroom is visible where the value is read. |
| 30 | + |
| 31 | +The other budgets are left alone: the Windows job at 15 minutes and the Python matrix at 33 to 65 minutes have not been observed to cancel, and nothing measured here applies to them. |
| 32 | + |
| 33 | +## Consequences |
| 34 | + |
| 35 | +Easier: a slow runner no longer turns `main` red, and a rerun stops being part of the merge routine. |
| 36 | + |
| 37 | +Harder: a genuinely hung SQLite suite now burns 20 minutes of runner time instead of 10 before the job fails. |
| 38 | + |
| 39 | +Unchanged: `CI Green` still requires this job to succeed, so a real failure blocks a merge exactly as before. |
| 40 | + |
| 41 | +What to watch: the p90 recorded here, 348 s. If a future measurement puts it near 600 s, the suite has grown into the new headroom and the answer is to split the job, not to raise the number again. |
0 commit comments