Skip to content

Commit 10b98d0

Browse files
cdeustclaude
andauthored
ci(test-sqlite): set the job budget from its measured p90 (#594) (#595)
The 10-minute budget left 1.7x headroom over this job's p90, and two runs of 2026-09-16 exceeded it: main at a981f93, cancelled at 616s having reached 74% of the suite, which made CI Green fail on test-sqlite='cancelled'; and PR #584's first attempt at 610s. Both passed on a rerun, in 416s and 276s, with no code change. The main run's own tree had taken 266s as the pull request of the commit it squashed. Measured over that day's 21 job executions, every attempt counted: 18 successes from 234s to 416s, median 328s, p90 348s, and three non-successes: the two timeouts above and a 47s cancellation from a superseded push. The budget is now 20 minutes, 3.4x that p90, about twice the point where both timeouts fired, and one eighteenth of GitHub's own ceiling, with the measurement recorded beside the number so suite growth into the new headroom stays visible (ADR-1074). The Windows and matrix budgets are untouched: nothing measured here applies to them. An earlier draft of this commit, its ADR and its CHANGELOG entry cited 29 runs, 27 successes and a second cancellation from a superseded push. That sample crossed several days and counted only each run's latest attempt, so it hid the two reruns and invented the 150s cancellation. The review of PR #595 caught it; the numbers above are the reproducible ones. Claude-Session: https://claude.ai/code/session_01Cz7A6goxVzfKL9NT7k4zDH Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 6a7e255 commit 10b98d0

5 files changed

Lines changed: 101 additions & 1 deletion

File tree

‎.github/workflows/ci.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -130,8 +130,14 @@ jobs:
130130
if: github.event_name != 'schedule' && (github.event_name != 'pull_request' || needs.changes.outputs.code == 'true' || needs.changes.outputs.deps == 'true' || needs.changes.outputs.workflows == 'true' || needs.changes.outputs.docs == 'true')
131131
name: Test (SQLite backend)
132132
runs-on: ubuntu-latest
133+
# 20 minutes is 3.4x the p90 of this job, measured over the 21 job
134+
# executions of 2026-09-16, every attempt counted: 18 successes from 234s
135+
# to 416s, median 328s, p90 348s. The former 10-minute budget left 1.7x
136+
# over that p90 and two runs exceeded it that day, main at a981f935
137+
# (cancelled at 616s) and PR 584's first attempt (610s); both passed on a
138+
# rerun. ADR number 1074 records this.
133139
# source: ADR-0802
134-
timeout-minutes: 10
140+
timeout-minutes: 20
135141

136142
# source: ADR-0802
137143

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,16 @@ adheres to [Semantic Versioning](https://semver.org/).
2828
for a `transcript_path` that is neither text nor a path, where it raised
2929
`TypeError`: the value comes from the hook event's untyped JSON envelope and
3030
runs in a path with no exception handler (ADR-1073).
31+
- **A slow runner no longer turns `main` red (#594).** The `Test (SQLite
32+
backend)` job's `timeout-minutes: 10` left 1.7x headroom over its measured
33+
p90, and two runs of 2026-09-16 exceeded it: `main` at `a981f935` (cancelled
34+
at 616 s, 74 % of the suite) and PR #584's first attempt (610 s). Both passed
35+
on a rerun, in 416 s and 276 s. Measured over that day's 21 job executions,
36+
every attempt counted: 18 successes from 234 s to 416 s, median 328 s, p90
37+
348 s, and three non-successes, the two timeouts plus a 47 s cancellation
38+
from a superseded push. The budget is now 20 minutes, 3.4x that p90 and
39+
about twice the point where both timeouts fired, with the measurement
40+
recorded beside the number (ADR-1074).
3141

3242
- **Procedural learning finally has an input (#591).** Every session entry was
3343
written with `toolsUsed: []` and `turnCount: 0`, because the SessionEnd
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
<!-- Generated by Cortex wiki_reindex from wiki/adr/cortex/1074-the-sqlite-test-job-s-budget-is-set-from-its-measured-p90.md; ADR-1074. Read-only mirror: edit the canonical wiki page. -->
2+
3+
---
4+
created: 2026-09-16T15:45:52Z
5+
kind: adr
6+
number: 1074
7+
status: accepted
8+
tags: [ci, workflows, timeout, issue-594]
9+
title: The SQLite test job's budget is set from its measured p90
10+
---
11+
# ADR-1074: The SQLite test job's budget is set from its measured p90
12+
13+
## Status
14+
15+
accepted
16+
17+
## Context
18+
19+
`main` went red at `a981f935`: the `Test (SQLite backend)` job hit its `timeout-minutes: 10` budget after 616 s, having reached 74 % of the suite, and `CI Green` failed on `test-sqlite='cancelled'` (run 35092942280). Re-running the failed jobs turned `main` green with no code change.
20+
21+
The same tree took 266 s as the pull-request run of `33dc284a`, the commit `a981f935` squashed, so one runner took at least 2.3× another for identical work. Measured over that day's 21 job executions, every attempt counted: 18 successes between 234 s and 416 s, median 328 s, p90 348 s, and three non-successes. Two of those three are timeouts, not contention noise: `main` at 616 s and PR #584's first attempt at 610 s, each passing on a rerun in 416 s and 276 s; the third is a 47 s cancellation from a superseded push. The slowest success used 69 % of the 600 s budget and the p90 used 58 %, so a runner three quarters slower than the p90 exceeds it, which happened twice in one day.
22+
23+
The budget exists to catch a hung suite before GitHub's own 6-hour ceiling (ADR-0802), not to police runner speed. At 1.7× the p90 it was doing the second.
24+
25+
The job runs the whole suite serially, `pytest --tb=short -q -p no:randomly`, on `ubuntu-latest`. Splitting it across workers would cut the wall clock, but the suite shares SQLite and PostgreSQL fixtures whose isolation under `pytest-xdist` is unproven here; that is a separate measurement, not a timeout decision.
26+
27+
## Decision
28+
29+
`Test (SQLite backend)` gets `timeout-minutes: 20`, which is 3.4× its measured p90, about twice the point where both timeouts fired, and still one eighteenth of GitHub's ceiling, and the workflow carries the measurement next to the number so that suite growth eating the new headroom is visible where the value is read.
30+
31+
The other budgets are left alone: the Windows job at 15 minutes and the Python matrix at 33 to 65 minutes have not been observed to cancel, and nothing measured here applies to them.
32+
33+
## Consequences
34+
35+
Easier: a slow runner no longer turns `main` red, and a rerun stops being part of the merge routine.
36+
37+
Harder: a genuinely hung SQLite suite now burns 20 minutes of runner time instead of 10 before the job fails.
38+
39+
Unchanged: `CI Green` still requires this job to succeed, so a real failure blocks a merge exactly as before.
40+
41+
What to watch: the p90 recorded here, 348 s. If a future measurement puts it near 600 s, the suite has grown into the new headroom and the answer is to split the job, not to raise the number again.
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
created: 2026-09-16T15:45:52Z
3+
kind: adr
4+
number: 1074
5+
status: accepted
6+
tags: [ci, workflows, timeout, issue-594]
7+
title: The SQLite test job's budget is set from its measured p90
8+
---
9+
# ADR-1074: The SQLite test job's budget is set from its measured p90
10+
11+
## Status
12+
13+
accepted
14+
15+
## Context
16+
17+
`main` went red at `a981f935`: the `Test (SQLite backend)` job hit its `timeout-minutes: 10` budget after 616 s, having reached 74 % of the suite, and `CI Green` failed on `test-sqlite='cancelled'` (run 35092942280). Re-running the failed jobs turned `main` green with no code change.
18+
19+
The same tree took 266 s as the pull-request run of `33dc284a`, the commit `a981f935` squashed, so one runner took at least 2.3× another for identical work. Measured over that day's 21 job executions, every attempt counted: 18 successes between 234 s and 416 s, median 328 s, p90 348 s, and three non-successes. Two of those three are timeouts, not contention noise: `main` at 616 s and PR #584's first attempt at 610 s, each passing on a rerun in 416 s and 276 s; the third is a 47 s cancellation from a superseded push. The slowest success used 69 % of the 600 s budget and the p90 used 58 %, so a runner three quarters slower than the p90 exceeds it, which happened twice in one day.
20+
21+
The budget exists to catch a hung suite before GitHub's own 6-hour ceiling (ADR-0802), not to police runner speed. At 1.7× the p90 it was doing the second.
22+
23+
The job runs the whole suite serially, `pytest --tb=short -q -p no:randomly`, on `ubuntu-latest`. Splitting it across workers would cut the wall clock, but the suite shares SQLite and PostgreSQL fixtures whose isolation under `pytest-xdist` is unproven here; that is a separate measurement, not a timeout decision.
24+
25+
## Decision
26+
27+
`Test (SQLite backend)` gets `timeout-minutes: 20`, which is 3.4× its measured p90, about twice the point where both timeouts fired, and still one eighteenth of GitHub's ceiling, and the workflow carries the measurement next to the number so that suite growth eating the new headroom is visible where the value is read.
28+
29+
The other budgets are left alone: the Windows job at 15 minutes and the Python matrix at 33 to 65 minutes have not been observed to cancel, and nothing measured here applies to them.
30+
31+
## Consequences
32+
33+
Easier: a slow runner no longer turns `main` red, and a rerun stops being part of the merge routine.
34+
35+
Harder: a genuinely hung SQLite suite now burns 20 minutes of runner time instead of 10 before the job fails.
36+
37+
Unchanged: `CI Green` still requires this job to succeed, so a real failure blocks a merge exactly as before.
38+
39+
What to watch: the p90 recorded here, 348 s. If a future measurement puts it near 600 s, the suite has grown into the new headroom and the answer is to split the job, not to raise the number again.

‎wiki/manifest.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3463,6 +3463,10 @@
34633463
"ADR-1073": {
34643464
"mirror": "ADR-1073-reading-a-transcript-has-one-implementation-and-refuses-a-non-path.md",
34653465
"path": "adr/cortex/1073-reading-a-transcript-has-one-implementation-and-refuses-a-non-path.md"
3466+
},
3467+
"ADR-1074": {
3468+
"mirror": "ADR-1074-the-sqlite-test-job-s-budget-is-set-from-its-measured-p90.md",
3469+
"path": "adr/cortex/1074-the-sqlite-test-job-s-budget-is-set-from-its-measured-p90.md"
34663470
}
34673471
},
34683472
"project": "cortex",

0 commit comments

Comments
 (0)