Repository navigation
ci: remove every time limit that can fail a run for being slow - #373
Conversation
…ule) Owner rule, 2026-10-12 15:52 (no time limits of any kind). Two runs that day failed ONLY because a bound was too tight for a slow row: - hosted kernel-only run 38213895073: the ci_verify ctest stage bound (1800 s) killed test_ci_verify, exit 124; - main push run 38217844683, native-live (native): "test_ci_verify: TIMEOUT 3600 must be positive and less than stage deadline 2820", i.e. main refused its own row set because the row bound outlived the matrix stage deadline. The cure is removal, not a bigger number. A hung job is caught only by the platform's own maximum (GitHub jobs: 360 minutes). What was removed: - scripts/ci_verify.py: the CTEST_TIMEOUT (1800) and SANITIZERS_FULL_CTEST_TIMEOUT (3600) constants; ctest_timeout() now returns cfg.ctest_stage_timeout, None unless --ctest-timeout is passed explicitly; every literal timeout= on a stage subprocess (23 call sites) and the timeout=600 defaults of invoke/call_runner/default_runner are gone, so subprocess.run runs unbounded; ctest_command always passes --timeout 21600 (the platform job maximum) unless --test-timeout overrides it, so CTest's undocumented built-in default can never decide a run; ctest_deadline_errors returns [] with no explicit stage deadline and the validate_config refusal needs both explicit flags; the summary fields testTimeoutSeconds/ctestTimeoutSeconds report null (no reader outside the verifier's own tests reads them); - scripts/ci_preflight.py: STAGE_TIMEOUT_SECONDS (2400) and the stage subprocess timeout (and the now-dead TimeoutExpired branch); JOB_RUNNERS pins every job at 360; NATIVE_STRATEGY loses the per-profile job_minutes/verify_minutes/test_timeout/ctest_timeout keys; - tests/CMakeLists.txt (10 sites) and runner/CMakeLists.txt (18 sites): every per-test TIMEOUT property removed; SKIP_RETURN_CODE, LABELS, ENVIRONMENT, add_test, row names and registration order untouched; the _pf_confirmed_refusal_timeout variable and the now-empty sanitizer if() blocks are gone with their comments; examples/native/CMakeLists.txt loses its example_* TIMEOUT 60 property and examples/native/run_example.cmake applies no timeout unless the caller passes -DTIMEOUT (its mutation self-test passes -DTIMEOUT=1 and still exercises the hang path); - .github/workflows/ci.yml, corpus-parity.yml, native-live.yml: every job-level and step-level timeout-minutes is 360 (11 sites, including the five-minute jobs); the native-live matrix keys job_minutes, verify_minutes, test_timeout and ctest_timeout are gone (the job and step use 360 directly) and the verify command no longer passes --test-timeout/--ctest-timeout; - tests: scripts/test_ci_verify.py and scripts/test_ci_preflight.py adapted (ctest_timeout is None for every profile, an explicit value still wins, ctest_command carries 21600 by default, the recorded runner sees timeout=None, no stage deadline refuses no row while an explicit one still does, every pinned job is 360 and a per-profile budget added back is refused). docs/ci.md rewritten sentences (quoted; the table rows below are rewritten: `preflight`, `build`, `sanitizers`, `kernel-only`, `native-live (native)`, `native-live (live-sanitizers)`, `native-live (live-tsan)`, `native-live-gate`, `corpus-parity`, `corpus-parity-subset`, `build` (aggregate), `changes`): - "The optional `--test-timeout N` gives CTest a default per-test deadline in seconds." -> "The opt-in `--test-timeout N` gives CTest a default per-test deadline in seconds and `--ctest-timeout N` sets the CTest stage's own deadline." - deleted: "Without these options, the driver's existing profile bounds and CTest behavior are unchanged." - deleted: "The native workflow supplies 2700/2820 seconds for native test/stage deadlines, and 900/1200 seconds for each live sanitizer." - "The summary records both bounds, discovered and selected row counts, executed rows, skipped or not-run rows, and stage timings, including when a test fails." -> the same with "both bounds (`null` when unset)". - "Their sanitizers CTest stage gets an hour (`ci_verify.py` bounds CTest at 30 minutes otherwise, the PR set included), because main's full set ran out of 30 minutes twice, and the sanitizers job allows 120 minutes for that hour after its build." -> "Their sanitizers CTest stage gets no bound of its own any more: main's full set ran out of the old 30-minute default twice, and the owner rule of 2026-10-12 removed the stage and row bounds rather than raising them again." - "Its sanitizer job ran 61.5 min and hit the verifier's 30 min CTest timeout." -> "... hit the verifier's then-30-minute CTest stage timeout (no stage carries a bound of ours any more)." - "The verifier's own bounds bind before these limits." and "By default, the driver gives CTest 30 minutes except for a full sanitizers run, which gets 60 minutes. The native workflow overrides these defaults with 47 minutes for native and 20 minutes for each live sanitizer profile. ... The row's growth (341-621 s a day earlier) is the next limit the whole CI meets, on any runner." -> "No bound of the verifier's own is left to bind first. ... Now no stage, row, step, profile or matrix entry carries a time limit of this repository's: every row runs until it finishes, and the only hang catcher is the platform's own 360-minute job maximum, which every job below carries." - "The limits below use measured durations where available and estimates for full sets on larger runners, with margins recorded in the table rather than a universal two-times multiplier. In particular, the native job's 75-minute limit is less than twice its observed 40.65-minute run; its row, CTest-stage and verification-step bounds leave the stated collection and setup margins." -> "The table below reports each job's measured duration and the platform maximum that replaced its old limit, so the margin the removal buys stays visible. The numbers are measured durations, not a two-times multiplier." - table cell "Each stage is bounded at 40 min (2400 s) inside it." -> "Its stages carry no timeout of their own any more -- the old 2400 s stage bound is gone (owner rule, 2026-10-12), so 360 minutes is all the room a slow verifier gets." Naming the deletions scripts/check_doc_reverts.py reports (introduced by 065ef1d "Split native CI profiles and make cold verification reproducible (#354)"), all replaced by the opt-in/no-bound text quoted above: - "Existing explicit `TIMEOUT` properties remain unchanged; the driver checks that every selected row's effective deadline is positive and shorter than the CTest stage deadline." - "`--ctest-timeout N` sets that stage's bound." - "Historically, `test_ci_verify` alone took up to 1288 s of the default 30 minutes at 0d76a09 (Ubuntu Debug); a fork's sanitizers compile took 1277-1335 s of the build stage's 30 minutes on the standard runner." No local suite was run (lane rule); py_compile, check_doc_anchors.py and check_doc_lint.py pass. ctest_command always passes --timeout: the explicit --test-timeout, else 21600 s (the platform's 360-minute job maximum). CTest's own default per-test timeout (1500 s, include(CTest)'s DART_TESTING_TIMEOUT, the figure the repository's comments used) would otherwise bind; this tree calls enable_testing() only, so the built-in default was not confirmed by a run; docs/ci.md and the ci_verify.py comment carry that caveat. The read-only review of this commit asked for it (the first wording stated 1500 s as fact). Also in this commit, beyond the verifier and workflow bounds: the eight subprocess bounds of scripts/check_native_cpp_abi.py (compile, link and version probes at 90/120/30 s, lines 468, 659, 835, 851, 870, 885, 898, 986: the thinnest margin on main's push CI, whose row test_native_cpp_abi takes 46-52 s on the hosted release and kernel runs and was not measured on the Debug and sanitizer legs), and the TIMEOUT 120 of the compile probe in tests/removed_spellings.cmake. Left in place for a second change (EN-105): the other harness-local subprocess timeouts under scripts/check_*.py, scripts/test_*.py, tests/*_e2e.py and docker/*_test.py. Co-Authored-By: Claude Code <[email protected]>
output_message_hash() seeded FNV-1a 64 with 1469598103934665603, the
standard offset basis 14695981039346656037 with its last digit dropped,
while pineforge.h and docs/outputs.md promise FNV-1a 64 and tell users
to re-hash the bytes they recorded. message_hash64 is new in 1.5.0, so
the standard basis costs nothing before the release.
The test helper fnv1a64() repeated the same truncated seed, so the
recorder test compared the bug with itself and could never catch it.
The recorder test now also pins the published FNV-1a 64 test vectors
("" -> 0xcbf29ce484222325, "a" -> 0xaf63dc4c8601ec8c), and
docs/outputs.md states the basis, the prime and the "a" vector.
The internal state-hash folds that share the old constant are not
public hashes and stay as they are.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|
…415) * ci: keep every verification job on standard runners Apply the owner cost stop of 2026-10-18 to pushes, schedules, dispatches and every pull request until the ci-spot pool is live. Replace the paid-runner routing introduced by d093560 and carried into native-live by 065ef1d; retain the standard image matrix and all existing timeouts. No tests, builds or workflows run locally per the executor brief. * ci: enforce the standard runner contract in preflight Replace the event-based runner contract and mutation cases introduced by d093560 with ubuntu-24.04 and matrix.os. Refuse pf-linux-x64-16, every *-xlarge or *-large label and every larger_runner key, including in the release matrix, while retaining the remaining workflow checks. Add rejection cases for paid labels, obsolete keys and conditional paid routing. Preserve comment tolerance and core-count checks. Tests intentionally not run under the executor no-run rule. * docs: record the standard runner cost stop and ci-spot handoff Reword the runner-selection and timing assumptions introduced by d093560 (#290) and the documentation-only runner sentences introduced by 133f571 (#308). Keep the platform job limits recorded by 2394ba2 (#373). Record the owner decision of 2026-10-18 after larger-runner costs exceeded US$3,000: every event uses free standard runners until the ci-spot pool is live. Describe the unconditional runner contract and remove obsolete larger-runner timing estimates while retaining measured standard-runner durations. No local verification run, as required by the executor brief. * test(ci): pin the standard-runner mutations precisely Target the full runs-on line in both build-runner rejection cases. The bare matrix.os expression matched the earlier leg name, leaving runs-on unchanged and producing the wrong finding in PR #415 CI run 39119916194. Preserve the build leg name and the expected runner-finding assertion. Leave scripts/ci_preflight.py unchanged. No tests or verification run locally under the executor no-run rule.
* ci: route CI Linux x64 jobs through the live ci-spot pool * ci: route corpus parity through the ci-spot build class * ci: route native live profiles through the ci-spot pool * ci: route documentation through the public ci-spot class * ci: route manual baseline promotion through ci-spot * ci: route release Linux x64 jobs through ci-spot * ci: require exact ci-spot routing and job labels in preflight Replace the CR-170 standard-only runner contract with the public and engine-x64-16 fragments, including fork and untrusted-event fallback. Require one matching PF_CI_LABELS string at job scope; pin the Linux-only build and release matrix routing and the unchanged hosted legs. Retain larger-runner, timeout, core-count and reusable-workflow guards. Update mutation and stage tests for missing, duplicate and incorrect PF_CI_LABELS, wrong label lists, unsafe fragment changes, paid labels and hosted exceptions. No tests, builds, linters or verification scripts run locally under the executor no-run rule. * docs: record the live ci-spot pool and hosted fallbacks Supersede the stopgap sentences from 3484a7c, squash-merged as 693fd9e (#415, CR-170). Reword the runner-selection and timing assumptions from d093560 (#290) and the documentation-only runner sentence from 133f571 (#308); preserve the platform limits from 2394ba2 (#373). Record the 2026-10-18 live-pool bulletin, exact fragment and environment labels, class allocation, actor allowlist, hosted matrix legs and event fallbacks. Keep historical timings separate from unmeasured pool performance. No local verification run under the executor no-run rule. * ci: restore release and deployment jobs to their original hosted runners Apply the L6 production-credential ruling to Cloudflare docs deployment, baseline promotion and every release job. Restore all three workflow files byte-for-byte from origin/main, including their original hosted labels, and remove pool environment labels. No steps, permissions, timeouts or matrix populations change. No tests, builds, linters, preflight, corpus verification or workflows run locally under the executor no-run rule. Nothing pushed. * ci: adopt the b240f6d7 hosted fallback for pool jobs Match the current pf-ci-spot public and engine-x64-16 fragments from bulletin #635. Change the hosted fallback to ubuntu-24.04 for all ten pool job definitions and preserve their matching PF_CI_LABELS, actor checks, fork protection and macOS routing. No tests, builds, linters, preflight, corpus verification or workflows run locally under the executor no-run rule. Nothing pushed. * ci: require hosted runners for credential-bearing jobs Pin the b240f6d7 expressions and original hosted release/deployment routes. Detect non-GITHUB_TOKEN secrets, inherited workflow environment secrets, job or workflow OIDC write permissions and deployment environments. Require hosted routing without PF_CI_LABELS for credentials and current fragments with matching labels for other Linux x64 jobs. Preserve the larger-runner, timeout and core-count guards. Add must-fail credential and old-fallback cases and a must-pass case for the current hosted release workflow. No tests, builds, linters, preflight, corpus verification or workflows run locally under the executor no-run rule. Nothing pushed. * docs: record the L6 hosted credential rule and current pool fallback Reword the pool-allocation, release/deploy routing and fallback sentences introduced by 28fe3466a and carried into f9bee22. Cite pf-ci-spot b240f6d7 and bulletin #635, retain the Dependabot and historical policy references, and explain the exact original hosted labels and updated preflight contract. No tests, builds, linters, preflight, corpus verification or workflows run locally under the executor no-run rule. Nothing pushed. * test(ci): anchor pool label mutations to job-level env An unanchored four-space env search matched the tail of step-level env in promotion and release jobs, so the negative fixture never put PF_CI_LABELS in the job environment. Anchor the env and steps keys to complete job-level lines in both label fixture builders. Assert the injected label at job scope and require the exact hosted-job diagnostic. Keep the existing hosted release baseline and the production guard unchanged. No tests, builds, linters or verification scripts run locally under the executor no-run rule. Nothing pushed.
…1ba62b) on core/cr150-o1-integ b4d2011 One commit carrying the tree of o1w/merge0 0a41361 (the full-history record of the merge-7 line), as a child of core/cr150-o1-integ b4d2011, so the doc-reverts gate judges the whole change against its real parent with this message (CORE-TOP ruling 10-21 18:04: no rewrite of pushed history, no checker exception). Merged lane heads (all carried in the tree): o1w-contract b19d8ae, o1w-pyjson cbc68d2, o1w-tail 27fcad9, o1w-selected 91fbd0e, o1w-seq 4b84f9d, o1w-tradeout bf024f1, o1w-ledger a87e546, o1w-metrics 3e1e70d, o1w-slabfold d41311c, batch-B 583c221, L8d proof 07c6363, L9b proof b01fe68, batch-C e6f8fcb, Q7 abort d220bd7, Q9 converter 3478350. Merge 7 is engine main 21ba62b. Rulings applied: CORE-TOP A (the four unbacked contract declarations withdrawn until the ledger and metrics lanes define them), B (the two _JSONRecordArray factory forms in the selected-window ownership scanner), E and F (closed-row accessor by copy; indexed report-array read), R13 (main's typed finite failures and proportional sub-day annualization win), the Option A failure-class split. DOC-REVERTS NAMES: every sentence deletion or revert listed by the doc-reverts gate on preflight 6f8e97 is a deliberate replacement made by an accepted ruling, a regeneration, or a later rewrite of the same page, and is named here by the commit that introduced it: - 03b2406 (ci: release jobs run on pf-ci-release, #461): the docs/ci.md paragraphs "Push this prepared migration only after both CORE-TOP gates", "Until then a job on that label with no runners only waits, and nothing is pushed.", the PF_CI_LABELS / pre-job hook / release.yml / promote-baseline.yml sentences: rewritten by main's own later edit of that page (21ba62b, #464), taken whole by merge 7. - a8f84b9 (docs(api): C ABI reference site): docs/pages/report-schema.md "pf_report_t is the single output of every backtest.", "This page maps each field to its meaning and units.", "Heap array, ordered by exit time ascending." rewritten by the O(1) report contract pages. - efb8c7e (o1w-contract round 2): docs/pages/report-schema.md "C/ctypes clients require an explicitly negotiated provider bridge; there is no new runtime C export in this slice." superseded by the selected-window named-sink contract text of b19d8ae. - 39a718b (cr150 L6b-1): docs/pages/run-failure-codes.md "The engine could not create or write the report sink." and "The run has no completed sink or valid trailer." replaced by the regenerated report_sink_failed description. - d220bd7 (cr150-q7): run-failure-codes.md "The existing engine_fault/retryable policy is retained pending a ruling:", "PROPOSED cancelled, retryable false;" and "RETURN TO CORE-TOP for the class ruling." removed by the class ruling (run_aborted only, no cancelled class, no retry field). - d91c96d (CR-150 route (ii) foundation): run-failure-codes.md "The engine could not create, resize, read or write scratch-table storage." replaced by the split wording (auxiliary tables raise scratch_io_failed; report-sink scratch raises report_sink_failed reason scratch_io). - e6f8fcb (CR-150 batch-C): docs/design/arena-bound.md "`sequence_index_` `include/pineforge/source/pine_adapter.hpp:N`" and the "`include/pineforge/detail/id_pair_store.hpp:N`;" citation: re-pointed or dropped after the merged source moved. - 5dbe91c (merge batch-B 583c221): docs/design/native-feature-parity.md "**yes** — `fill_report` engine_report.cpp:N, `get_trade` engine.hpp:N, `compute_trade_stats`" and "**no** — `compute_equity_stats` exits early on an empty curve": single-line pins replaced by the backticked symbol, as the anchor checker instructs. - 4d6cd2b (CR-150 WL-PTAILFIX): docs/design/consumed-partial-exit-fold.md "The previous-header fail-before compilation of the new unit remains required; its expected first diagnostic is the additional `IdPairStore` template a..." rewritten by o1w-slabfold d41311c. - d093560 (INT26 wave H): docs/pages/native-engine.md "A C++ host reads a closed row whole, at any time, every `Trade` field included." replaced by the closed-row-access-by-copy row (CORE-TOP ruling E: closed rows are paged, the C++ accessor returns a copy). DOC-REVERTS NAMES, second set: against this commit's real parent b4d2011 the gate reports 21 unnamed sentence deletions or reverts, all introduced by the seven commits below (preflight rj-20261021t100903-91b0cc, stage doc-reverts). Each is a deliberate rewrite on top of the introducing commit, and each introducer is named here: - 133f571 ("1.0 prep: CI flake fix, docs-only CI skip, benchmark refresh (supersedes #303, #304, #307)"), eleven docs/ci.md sentences: "The compilation lanes start alongside preflight, as soon as the few-second `changes` job has classified the change", "A pull request or a push to `main` whose change is documentation only runs `preflight` and the documentation build", "Each proof job waits for `changes` alone, a few seconds once a standard runner takes it", "`preflight` waits for nothing.", "`changes` and `preflight` wait for nothing and run on every event", "A documentation-only change starts preflight alone there.", "So does the `changes` job, a two-commit checkout and one `git diff`.", "The same stage pins the documentation-only skip", "The advisory `build` aggregate passes a proof job that succeeded", "A proof job skipped for any other reason fails it", "success": rewritten by main's later CI redesign and carried whole by merge 7 of 21ba62b. - b2a578c ("docs: make every README and doc page true before the release (DOCS-ENGINE) (#300)"), one docs/ci.md sentence on the PR registration floors after lane FIX-E1E2 (773 Debug and sanitizers, 782 native): superseded by the later floor rows. - 2394ba2 ("ci: remove every time limit that can fail a run for being slow (#373)"), two docs/ci.md sentences: "The table below reports each job's measured duration and the platform maximum that replaced its old limit" and "The numbers are measured durations, not a two-times multiplier.": replaced by the current duration table. - 7da487a ("ci/tests: remove the CI time limits; a platform-maximum stop is a HANG to re-run (EN-116)"), two docs/ci.md sentences: "Under the owner rule, no required-check stage, row, step, profile or matrix entry may carry a time limit" and "The native workflow retains only its 360-minute job envelopes, with no verification-step deadline.": superseded by the same redesign. - b148979 ("selected window: declare 1.3-parity, not 1.3, until a completeness proof exists (1.6.1)"), three sentences, README.md and docs/pages/report-schema.md: "`capabilities.selected_window_contract_revisions` is `[\"1.3-parity\"]` when the runtime supports selected windows", "The `--capabilities-json` probe's `capabilities.selected_window_contract_revisions` is `[\"1.3-parity\"]`", "From 1.6.1 (unreleased), the image declares `1.3-parity` instead": reverted to the 1.3 pin by the later capability ruling (AMD6 1.3-parity only), an accepted reversal. - 1bd075c ("Trading metrics + equity curve in backtest report (ABI v2) (#31)"), one docs/pages/report-schema.md sentence: "See the per-field doxygen in `<pineforge/pineforge.h>` for the exact rule on every field.": replaced by the O(1) report contract's per-field pages. - d3cca9e ("Docs: correct migration semantics and harden publication guards"), one docs/design/native-feature-parity.md pin line (`sha256:H` c_abi.cpp:N, `sha256:H` check_c_abi_runtime.py:N, ...): replaced by the symbol-only rows, as the anchor checker instructs. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
No check or test fails for being slow any more (owner rule, 2026-10-12). Time limits that could turn a run red are removed, not raised. The only hang catcher left is the platform's job maximum.
.github/workflows/{ci,corpus-parity,native-live}.yml: every job'stimeout-minutesis 360, the GitHub-hosted maximum.native-livepasses no test or stage deadline.scripts/ci_verify.py: no ctest stage timeout for any profile. The 1800/3000/3600 s constants and the stages' literal subprocess bounds are gone.ctestgets--timeout 21600, so that no CTest built-in default per-test timeout can decide a run; on GitHub the 360-minute job cap comes first.--test-timeoutand--ctest-timeoutremain as opt-in flags, and the deadline check still refuses a row that is not strictly inside an explicit deadline when both are given.scripts/ci_preflight.py: the stage bound and the job-timeout pins follow the workflows.scripts/test_ci_verify.pyandscripts/test_ci_preflight.pyare adapted; each replaced test fails against the old code.scripts/check_native_cpp_abi.py: its compile and link subprocess bounds are removed.tests/CMakeLists.txt,tests/removed_spellings.cmake,runner/CMakeLists.txt,examples/native/: every per-testTIMEOUTis removed.docs/ci.md: updated to match.Why now: on main after #371, the hosted
native-live (native)failedctest-deadlines(thetest_ci_verifyrowTIMEOUT 3600against the 2820 s native stage) andkernel-onlyhit its 1800 s stage bound (#371 comment). Pull-request runs exclude slow rows, so both showed only on the push run.Proof: hosted CI on this PR (
kernel-onlyruns every row), a manualnative-livedispatch on this branch (slow rows included),pineforge/verifyfor the steps this diff can fail, andpineforge/parityfrom a recorded sweep of this tree. The parity diff rule does not apply, because the CMake files change.Not in this PR (tomorrow, separately): the subprocess
timeout=bounds in about 40 Python test and check files.🤖 Generated with Claude Code