Skip to content

Commit 2394ba2

Browse files
authored
ci: remove every time limit that can fail a run for being slow (#373)
Remove the ctest stage bounds in ci_verify and preflight, every per-test CMake TIMEOUT, and the native-live and kernel ctest deadlines; every job's timeout-minutes is now the platform maximum (360) as the only hang catcher. Also fix the seed of message_hash64 (new in 1.5.0) to the standard FNV-1a 64 offset basis, with a published test vector; the old test helper repeated the truncated seed, so it could not catch the bug. Verified on tree 399beb1: pineforge/verify 8/8 (kernel 337 rows in 1504 s with no stage bound) and pineforge/parity (sweep lvb20-ntl3-parity-20261012, no regression). One native_live_startup_e2e failure was a test ordering race, explained in the PR comment, passed on rerun and in the slow-row dispatch; the order-insensitive test fix follows.
1 parent b97c409 commit 2394ba2

18 files changed

Lines changed: 387 additions & 326 deletions

‎.github/workflows/ci.yml‎

Lines changed: 25 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,10 @@ env:
3838
# bills the organization even for a public repository, so every other event --
3939
# a fork's pull request above all -- runs on the free standard runner.
4040
# Each verification sizes its parallelism to the runner that took it.
41-
# scripts/ci_preflight.py pins every runner, time limit and job count, and
42-
# docs/ci.md gives each limit's measured basis.
41+
# scripts/ci_preflight.py pins every runner, time limit and job count; every
42+
# job's limit is the platform's own 360-minute maximum, the only hang catcher
43+
# (owner rule, 2026-10-12), and docs/ci.md records why no bound of our own
44+
# remains.
4345
jobs:
4446
# Whether the change is documentation only; scripts/ci_docs_only.py holds the
4547
# rule, and every doubt answers no. On a docs-only pull request or push the
@@ -49,7 +51,7 @@ jobs:
4951
# runner, for every event.
5052
changes:
5153
runs-on: ubuntu-24.04
52-
timeout-minutes: 5
54+
timeout-minutes: 360
5355
outputs:
5456
docs_only: ${{ steps.classify.outputs.docs_only }}
5557
steps:
@@ -78,11 +80,12 @@ jobs:
7880
7981
preflight:
8082
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
81-
# Its verifier-tests stage is one serial process: 475 s at 0d76a099's tree
82-
# on the maintainers' verification hosts, which ran it 1.93-1.95 times as
83-
# fast as the standard runner on the same trees, so about 15 min here. Ten
84-
# minutes cancelled main's run at 0d76a099 593 s into it.
85-
timeout-minutes: 45
83+
# 360 minutes is GitHub's job maximum and deliberately the only limit:
84+
# its verifier-tests stage is one serial process, and no stage of this
85+
# preflight carries a timeout of its own any more (owner rule,
86+
# 2026-10-12). A hang reaches the platform maximum; a slow stage never
87+
# fails the run.
88+
timeout-minutes: 360
8689
steps:
8790
- uses: actions/checkout@v4
8891
with:
@@ -140,11 +143,11 @@ jobs:
140143
# accepts; a failed or unanswered classification runs it.
141144
needs: changes
142145
if: ${{ !cancelled() && (needs.changes.result != 'success' || needs.changes.outputs.docs_only != 'true') }}
143-
# A full-set leg waits on the test_ci_verify CTest row, 976-1288 s on the
144-
# standard runners at 0d76a099, which more cores do not shorten. There the
145-
# Release legs, which a fork's pull request runs in full, took 30.3 and
146-
# 32.0 min, and the macOS Debug leg 37.0 min.
147-
timeout-minutes: 75
146+
# 360 minutes (the platform maximum): a full-set leg waits on the
147+
# test_ci_verify CTest row, 976-1288 s on the standard runners at
148+
# 0d76a099, which more cores do not shorten. No bound of ours may fail a
149+
# slow leg (owner rule, 2026-10-12).
150+
timeout-minutes: 360
148151
strategy:
149152
fail-fast: false
150153
matrix:
@@ -238,11 +241,11 @@ jobs:
238241
needs: changes
239242
if: ${{ !cancelled() && (needs.changes.result != 'success' || needs.changes.outputs.docs_only != 'true') }}
240243
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
241-
# A full run (push to main, manual dispatch) gives its CTest stage an hour
242-
# after the build; the PR set keeps its 30-minute stage, after about 30
243-
# minutes of build and ABI providers on the standard runner a fork's pull
244-
# request gets.
245-
timeout-minutes: 120
244+
# 360 minutes (the platform maximum). No CTest stage bound remains: main's
245+
# full set ran out of the old 30- and 60-minute stage bounds twice, and
246+
# the owner rule of 2026-10-12 removed the bounds rather than raising
247+
# them again.
248+
timeout-minutes: 360
246249
env:
247250
ASAN_OPTIONS: detect_leaks=1:halt_on_error=1:abort_on_error=1
248251
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
@@ -303,8 +306,9 @@ jobs:
303306
if: ${{ !cancelled() && (needs.changes.result != 'success' || needs.changes.outputs.docs_only != 'true') }}
304307
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
305308
# 22.3 min on the standard runner at 0d76a099, where a fork's pull request
306-
# still runs the whole kernel set.
307-
timeout-minutes: 60
309+
# still runs the whole kernel set; 360 minutes is the platform maximum, the
310+
# only hang catcher (owner rule, 2026-10-12).
311+
timeout-minutes: 360
308312
steps:
309313
- uses: actions/checkout@v4
310314
with:
@@ -384,7 +388,7 @@ jobs:
384388
if: always()
385389
# A few seconds of shell: the standard runner, for every event.
386390
runs-on: ubuntu-24.04
387-
timeout-minutes: 5
391+
timeout-minutes: 360
388392
steps:
389393
- name: Require every verification lane to have passed
390394
shell: bash

‎.github/workflows/corpus-parity.yml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -92,8 +92,10 @@ jobs:
9292
corpus-parity:
9393
if: inputs.mode != 'subset'
9494
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
95-
# 16.3-26.2 min on the standard runner a fork's pull request still gets.
96-
timeout-minutes: 120
95+
# 16.3-26.2 min on the standard runner a fork's pull request still gets;
96+
# 360 minutes is the platform maximum, the only hang catcher (owner rule,
97+
# 2026-10-12: no bound of our own may fail a run for slowness).
98+
timeout-minutes: 360
9799
steps:
98100
- uses: actions/checkout@v4
99101
with:
@@ -167,8 +169,9 @@ jobs:
167169
corpus-parity-subset:
168170
if: inputs.mode == 'subset'
169171
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
170-
# 1.8-4.7 min on the standard runner.
171-
timeout-minutes: 30
172+
# 1.8-4.7 min on the standard runner; 360 minutes is the platform
173+
# maximum, the only hang catcher (owner rule, 2026-10-12).
174+
timeout-minutes: 360
172175
steps:
173176
# Credentials stay persisted, exactly as the nightly job above: the
174177
# submodule clone and `git lfs pull` below go out over https and the

‎.github/workflows/native-live.yml‎

Lines changed: 11 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -40,36 +40,24 @@ jobs:
4040
name: native-live (${{ matrix.profile }})
4141
# Same trusted-event 16-core policy as CI; forks use the standard runner.
4242
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)) && 'pf-linux-x64-16' || 'ubuntu-24.04' }}
43-
timeout-minutes: ${{ matrix.job_minutes }}
43+
timeout-minutes: 360
4444
strategy:
4545
fail-fast: false
4646
matrix:
4747
include:
4848
- profile: native
4949
build_dir: build-live
50-
job_minutes: 75
51-
verify_minutes: 65
52-
test_timeout: 2700
53-
ctest_timeout: 2820
5450
- profile: live-sanitizers
5551
build_dir: build-live-sanitizers
56-
job_minutes: 45
57-
verify_minutes: 35
58-
test_timeout: 900
59-
ctest_timeout: 1200
6052
- profile: live-tsan
6153
build_dir: build-live-tsan
62-
job_minutes: 45
63-
verify_minutes: 35
64-
test_timeout: 900
65-
ctest_timeout: 1200
66-
# Main runs 37398893123/37461653250/37478400505 took 35.4–40.7m
67-
# native + 15.1–16.3m ASan before TSan exhausted the shared hour, despite
68-
# compiler and curl cache hits. Independent 75/45/45m budgets allow cold
69-
# builds and leave ten minutes outside verification for setup/collection.
70-
# test_ci_verify took 1695s on GitHub and 2178s on a c5.4xlarge; 2700s
71-
# gives that host 24% margin, inside a 2820s stage. Live rows allow up to
72-
# 900s; keep those limits and a 1200s stage. No row inventory changes.
54+
# 360 minutes is GitHub's job maximum. It is deliberately the ONLY time
55+
# limit here (owner rule, 2026-10-12): the old 75/45/45-minute budgets and
56+
# the verifier's 2700/2820 and 900/1200-second row and stage deadlines
57+
# cut off runs whose own rows were merely slow -- main's 2820s stage
58+
# deadline refused this very row set's 3600s TIMEOUT property -- so no
59+
# stage, step or row carries a bound of ours any more. Only a hang reaches
60+
# the platform maximum. No row inventory changes.
7361
env:
7462
CCACHE_DIR: ${{ github.workspace }}/.ccache/${{ matrix.profile }}
7563
steps:
@@ -121,8 +109,8 @@ jobs:
121109
122110
- name: Verify (${{ matrix.profile }})
123111
id: verify
124-
timeout-minutes: ${{ matrix.verify_minutes }}
125-
run: python3 scripts/ci_verify.py ${{ matrix.profile }} --build-dir ${{ matrix.build_dir }} --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache --test-timeout ${{ matrix.test_timeout }} --ctest-timeout ${{ matrix.ctest_timeout }} ${{ matrix.profile == 'native' && '--require-websocket' || '' }} ${{ matrix.profile == 'native' && inputs.exclude_slow && '--exclude-label slow' || '' }}
112+
timeout-minutes: 360
113+
run: python3 scripts/ci_verify.py ${{ matrix.profile }} --build-dir ${{ matrix.build_dir }} --jobs "$(getconf _NPROCESSORS_ONLN)" --generator Ninja --curl-dir "${{ steps.curl-deps.outputs.curl-dir }}" --ccache ${{ matrix.profile == 'native' && '--require-websocket' || '' }} ${{ matrix.profile == 'native' && inputs.exclude_slow && '--exclude-label slow' || '' }}
126114

127115
- name: Stage and summarize diagnostics
128116
if: always()
@@ -159,7 +147,7 @@ jobs:
159147
needs: native-live
160148
if: always()
161149
runs-on: ubuntu-24.04
162-
timeout-minutes: 5
150+
timeout-minutes: 360
163151
steps:
164152
- name: Require every native profile to have passed
165153
env:

0 commit comments

Comments
 (0)