Repository navigation
ci: keep every CI round under 15 minutes - #849
luisleo526 wants to merge 3 commits into
Conversation
Every CI round, post-merge included, now takes 15 minutes or less. GitHub
CI keeps smoke legs on every configuration; the full profiles leave GitHub
CI for the lab's verify, which every merge already requires on the exact
head tree (pineforge/verify).
Measured: post-merge run 43205609467 took 123.5 minutes (32 of 49 jobs over
15 minutes); each full-profile leg rebuilt the engine from a cold compiler
cache in 11-40 minutes on a standard 4-vCPU runner, and 27 single CTest rows
ran over 15 minutes on their own. PR run 43242369382 took 75.3 minutes, 73.9
of them in build-arm's unsharded ci_verify.py debug step. The parity subset
step ran 13.0 and 13.75 minutes in those two runs: after the runtime it
built its 54 strategy libraries one at a time (6 min 53 s), because targets
named one by one on a build command line are built serially by a Makefile
generator.
- ci-post.yml: authz, changes, build-x86-gcc133, preflight (fast tier),
smoke and smoke-arm (Release and Debug on ubuntu-24.04 and
ubuntu-24.04-arm: pineforge_pr_smoke and ctest -L '^fast$'), sanitizers
(the same smoke under ASan and UBSan), corpus-parity-subset and the build
aggregate. Removed: preflight-slow, the shard matrices and their
aggregates, the native-live call, the full corpus-parity call and the
cold_native_cache input.
- gke-pr.yml: build-arm runs the Debug smoke on ubuntu-24.04-arm.
- corpus-parity.yml: no schedule; workflow_dispatch and workflow_call stay.
- scripts/check_corpus_parity.sh --subset and CMakeLists.txt: the subset's
strategy libraries build as the dependencies of one aggregate target
(pineforge_corpus_parity_subset), JOBS at a time; a target the corpus does
not define stops the generation.
- scripts/ci_preflight.py pins the new shape; scripts/ci_release_gate.py
requires the four smoke check names; the tests follow.
- docs/ci.md ("Where the full suites run"), AGENTS.md and CLAUDE.md: where
each cut suite now runs, the suites the lab does not run yet and the
interim rule for them, and the release rule.
Doc reverts, each replaced by the new shape's text in docs/ci.md:
- 41f08d2: "Complete Release, Debug, native, kernel and parity profiles plus
the required x86 build" (the ci-post.yml row of the trigger map).
- f85ce18: "Full parity and its post-merge subset" (the corpus-parity.yml
row of the trigger map).
- 3a0d150: "Scheduled and manual runs execute every profile." and
"Schedule, dispatch, post CI callers" (the tiers paragraph and the
corpus-parity.yml row of the workflow table).
- 8154993: "Dispatch and post CI callers on `main`" (the native-live.yml row
of the workflow table).
- 3ea5a0d: "All three build jobs refuse an OS mismatch.", "Every shard builds
the full tree and runs `--ctest-shard I/N` at the runner's core count." and
"Each build refuses an OS mismatch before checkout." (the post build jobs
are smoke legs now; smoke and smoke-arm refuse a machine mismatch).
- e7fa559: "Release uses four `build` shards, Debug two `build-debug` shards,
and kernel two shards; sanitizers uses four, all on the pinned hosted
images." (no post shard matrix remains).
Co-Authored-By: Claude Fable 5.2 <[email protected]>
Main gained "ci: retain PR smoke test output" (#800): a final collector step on the build and build-debug jobs of gke-pr.yml, and its pin tests. Both files merged without a conflict: - .github/workflows/gke-pr.yml: main's two collector steps (build and build-debug) and this branch's arm64 smoke steps in build-arm are separate hunks; build-arm keeps its five smoke steps. - scripts/test_ci_preflight.py: main's GkeSmokeOutputAdmission changes (the shipped-retention pin and its test) and this branch's changes (the smoke-leg tests, and the same class's setUp reading the legacy fixture through legacy_gke_arm_smoke) are separate hunks. With main's collector, the Debug smoke is no longer the last step of build-debug, while it stays the last step of build-arm. The branch's test that holds build-arm's steps to build-debug's therefore compares the two step blocks without the blank lines that end a job (changed in the branch's own commit, not in this merge). Co-Authored-By: Claude Fable 5.2 <[email protected]>
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Replaces the sharded full-profile build matrices in ci-post.yml with a single smoke job per configuration (Release and Debug) that builds the PR smoke target and runs the fast CTest label, so each round fits fifteen minutes. The full profiles and the slow preflight tier move out of the PR workflow to the lab's verify and preflight steps on the exact head tree before merge. The smoke job's verify step carries no command or action, the arm smoke job's checkout may leave the token available to build code, and dropping the cold_native_cache dispatch input removes the manual cold-cache run.
Worth a look
- Smoke verification step has no command or action —
.github/workflows/ci-post.yml:276· Escalate · high- unverified hypothesis — survived the adversarial refute pass; needs human review
- Checkout token remains available to smoke-arm build code —
.github/workflows/ci-post.yml· Escalate · medium- unverified hypothesis — survived the adversarial refute pass; needs human review
- Arm CI no longer runs the full verification profiles —
.github/workflows/ci-post.yml· Escalate · medium- unverified hypothesis — survived the adversarial refute pass; needs human review
- Fast CTest rows may run without their pytest dependency —
.github/workflows/ci-post.yml· Escalate · medium- unverified hypothesis — survived the adversarial refute pass; needs human review
- Workflow dispatch input removed —
.github/workflows/ci-post.yml:9· Escalate · medium- unverified hypothesis — survived the adversarial refute pass; needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2097 functions depend on the 2034 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
ci_workflow_findings()— 91 callers, 25 callees - new:
post_smoke_workflow_findings()— 1 callers, 6 callees
Verification — 2097 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2065 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
1425 of 1425 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety (
noncode-change,uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.
benchmarks/tests/test_aggregate.py— full-run-safetybenchmarks/tests/test_check_provenance.py— full-run-safetybenchmarks/tests/test_compare.py— full-run-safetybenchmarks/tests/test_public_names.py— full-run-safetybenchmarks/tests/test_run_all.py— full-run-safetybenchmarks/tests/test_run_strategy_feeds.py— full-run-safetydocker/pineforge_sink_finalize_test.py— full-run-safetydocker/pineforge_sink_replay_rq_test.py— full-run-safetydocker/pineforge_sink_replay_test.py— full-run-safetydocker/pineforge_sink_to_json_test.py— full-run-safetydocker/quota_columns_test.py— full-run-safetydocker/quota_convert_test.py— full-run-safetydocker/quota_finalize_test.py— full-run-safetydocker/replay_rq_engine_test.py— full-run-safetydocker/run_json_artifact_window_test.py— full-run-safetydocker/run_json_bin_test.py— full-run-safetydocker/run_json_bounded_transport_test.py— full-run-safetydocker/run_json_bulk_equity_fuzz_test.py— full-run-safetydocker/run_json_bulk_equity_test.py— full-run-safetydocker/run_json_codes_test.py— full-run-safetydocker/run_json_converter_metadata_test.py— full-run-safetydocker/run_json_converter_session_test.py— full-run-safetydocker/run_json_diagnostics_test.py— full-run-safetydocker/run_json_equity_scan_test.py— full-run-safetydocker/run_json_fused_test.py— full-run-safetydocker/run_json_harnesses_test.py— full-run-safetydocker/run_json_metrics_only_test.py— full-run-safetydocker/run_json_misc2_test.py— full-run-safetydocker/run_json_misc_test.py— full-run-safetydocker/run_json_pfbar_test.py— full-run-safetydocker/run_json_provenance_test.py— full-run-safetydocker/run_json_quota_supervisor_test.py— full-run-safetydocker/run_json_replay_test.py— full-run-safetydocker/run_json_selected_sink_test.py— full-run-safetydocker/run_json_sink_lexical_test.py— full-run-safetydocker/run_json_sink_summary_test.py— full-run-safetydocker/run_json_sink_test.py— full-run-safetydocker/run_json_sink_vectors_test.py— full-run-safetydocker/run_json_spool_work_test.py— full-run-safetydocker/run_json_stringio_test.py— full-run-safetydocker/run_json_symbol_feeds_test.py— full-run-safetydocker/run_json_syminfo_test.py— full-run-safetydocker/run_json_trade_scan_test.py— full-run-safetydocker/run_json_window_budget_test.py— full-run-safetydocker/run_json_wrapper_work_test.py— full-run-safetyinclude/pineforge/native_run_spec.hpp— full-run-safetyscripts/derive_corpus_feeds_self_test.py— full-run-safetyscripts/fingerprint_self_test.py— full-run-safetyscripts/run_corpus_self_test.py— full-run-safetyscripts/test_abi_receipt_skips.py— full-run-safety- … and 1375 more
non-code file(s) changed (
.github/workflows/ci-post.yml,.github/workflows/corpus-parity.yml,.github/workflows/gke-pr.yml,AGENTS.md,CLAUDE.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
.github/workflows/ci-post.yml,.github/workflows/corpus-parity.yml,AGENTS.md,CLAUDE.md,docs/ci.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
CHANGELOG.md§ Unreleased (lines 8-244): references changed symbolspostCHANGELOG.md§ CI and release process (lines 832-893): references changed symbolspostdocker/README.md§ Two-phase run (lines 98-274): references changed symbolspostdocker/README.md§ Opt-in supervised report channel (lines 356-398): references changed symbolspostdocs/pine-adapter-kernel-notes.md§ 2.2 Same-bar strategy.close batching (lines 749-780): references changed symbolspostdocs/pine-adapter-kernel-notes.md§ 4. Other orphaned fill / bracket / margin notes (lines 819-1024): references changed symbolspostdocs/reference/README.md§ Reference text moved out of CLAUDE.md and AGENTS.md (lines 1-12): references changed symbolspostdocs/reference/ci-runner-routing.md§ (document) (lines 1-11): references changed symbolspost
· 1 grounded finding(s) anchored inline below; 1 more finding(s) on lines outside this diff (see the check run).
|
|
||
| def post_shard_workflow_findings(jobs: dict[str, str]) -> list[str]: | ||
| """Keep each leg independent, its cache single-writer and diagnostics distinct.""" | ||
| def post_smoke_workflow_findings(jobs: dict[str, str]) -> list[str]: |
There was a problem hiding this comment.
post_smoke_workflow_findings()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
The change that moved the full profiles out of GitHub CI left sentences in docs/ci.md and docs/reference/ci-runner-routing.md that still described the old post-merge workflow. They now say what runs: smoke legs in ci-post.yml, the full profiles and the full parity in the lab's verify, no preflight-slow job, and the measured time of the parity subset job (6.8 minutes in run 43308686886, against 13.5 and 14.3 before its strategy libraries built in parallel). docs/ci.md keeps its line count. Two comments in scripts/test_ci_preflight.py call a CI leg a leg. Doc reverts, each replaced by a sentence about the new shape: - "`preflight-slow` runs exactly those stages with `--tier slow`" and "Toolchain setup and the hash-pinned pytest install stay in the slow job." (no GitHub job runs the slow tier; the lab's preflight step runs every tier). - "the caller's `build` aggregate still requires the reusable workflow to succeed" (ci-post.yml no longer calls native-live.yml). - "retains the subset through `corpus-parity.yml` (`mode: subset`) alongside its full-parity call" (there is no full-parity call). - "The GKE Linux arm64 Debug leg runs after merge; a red result there blocks the next release cut under the same fix-or-revert rule." and "The GKE Linux arm64 Debug leg also runs before merge when `arm_relevant=true`; a red post-merge result blocks the next release cut under the same rule." (the leg is a smoke now; the full arm64 Debug profile left GitHub CI). - "the full-profile commands in `ci-post.yml` carry no explicit `--exclude-label` arguments" and "PR CI keeps the Release smoke selected by CTest label `fast`." (the commands run in the lab's verify; GitHub CI keeps the smoke on every configuration). - "Pull requests run the Ubuntu amd64 Release smoke; post-merge runs Ubuntu amd64 and arm64 Release/Debug." and "Both architectures use the same full-profile commands." (the `build`, `build-arm` row of the job table). - docs/reference/ci-runner-routing.md: "Post-merge `ci-post.yml` runs the full profiles on x86 and arm64, plus native and parity callers; there is no macOS cell." and "the post-merge legs (`ci-post.yml`) run their full profile populations, and a documentation-only PR runs admission, preflight and the docs build while skipping proof legs." Co-Authored-By: Claude Fable 5.2 <[email protected]>
One commit on main 2bbdd48 with the tree of the five members merged in order: #849 (5c9bfc4), #846 (fdbe3b4), #845 (434cdc5), #847 (e04ccd2), #842 (c33724c). #842 carries #841: #841's three commits are ancestors of #842's head. == #849 == Every CI round, post-merge included, now takes 15 minutes or less. GitHub CI keeps smoke legs on every configuration; the full profiles leave GitHub CI for the lab's verify, which every merge already requires on the exact head tree (pineforge/verify). Measured: post-merge run 43205609467 took 123.5 minutes (32 of 49 jobs over 15 minutes); each full-profile leg rebuilt the engine from a cold compiler cache in 11-40 minutes on a standard 4-vCPU runner, and 27 single CTest rows ran over 15 minutes on their own. PR run 43242369382 took 75.3 minutes, 73.9 of them in build-arm's unsharded ci_verify.py debug step. The parity subset step ran 13.0 and 13.75 minutes in those two runs: after the runtime it built its 54 strategy libraries one at a time (6 min 53 s), because targets named one by one on a build command line are built serially by a Makefile generator. Run 43308686886 of the new ci-post.yml took 9 min 39 s end to end, queue included; its parity subset job took 6.8 minutes. - ci-post.yml: authz, changes, build-x86-gcc133, preflight (fast tier), smoke and smoke-arm (Release and Debug on ubuntu-24.04 and ubuntu-24.04-arm: pineforge_pr_smoke and ctest -L '^fast$'), sanitizers (the same smoke under ASan and UBSan), corpus-parity-subset and the build aggregate. Removed: preflight-slow, the shard matrices and their aggregates, the native-live call, the full corpus-parity call and the cold_native_cache input. - gke-pr.yml: build-arm runs the Debug smoke on ubuntu-24.04-arm. - corpus-parity.yml: no schedule; workflow_dispatch and workflow_call stay. - scripts/check_corpus_parity.sh --subset and CMakeLists.txt: the subset's strategy libraries build as the dependencies of one aggregate target (pineforge_corpus_parity_subset), JOBS at a time; a target the corpus does not define stops the generation. - scripts/ci_preflight.py pins the new shape; scripts/ci_release_gate.py requires the four smoke check names; the tests follow. - docs/ci.md ("Where the full suites run"), docs/reference/ci-runner-routing.md, AGENTS.md and CLAUDE.md: where each cut suite now runs, the suites the lab does not run yet and the interim rule for them, and the release rule. Doc reverts, each replaced by the new shape's text in docs/ci.md: - 41f08d2: "Complete Release, Debug, native, kernel and parity profiles plus the required x86 build" (the ci-post.yml row of the trigger map). - f85ce18: "Full parity and its post-merge subset" (the corpus-parity.yml row of the trigger map). - 3a0d150: "Scheduled and manual runs execute every profile." and "Schedule, dispatch, post CI callers" (the tiers paragraph and the corpus-parity.yml row of the workflow table). - 8154993: "Dispatch and post CI callers on `main`" (the native-live.yml row of the workflow table). - 3ea5a0d: "All three build jobs refuse an OS mismatch.", "Every shard builds the full tree and runs `--ctest-shard I/N` at the runner's core count." and "Each build refuses an OS mismatch before checkout." (the post build jobs are smoke legs now; smoke and smoke-arm refuse a machine mismatch). - e7fa559: "Release uses four `build` shards, Debug two `build-debug` shards, and kernel two shards; sanitizers uses four, all on the pinned hosted images." (no post shard matrix remains). - "`preflight-slow` runs exactly those stages with `--tier slow`" and "Toolchain setup and the hash-pinned pytest install stay in the slow job." (no GitHub job runs the slow tier; the lab's preflight step runs every tier). - "the caller's `build` aggregate still requires the reusable workflow to succeed" (ci-post.yml no longer calls native-live.yml). - "retains the subset through `corpus-parity.yml` (`mode: subset`) alongside its full-parity call" (there is no full-parity call). - "The GKE Linux arm64 Debug leg runs after merge; a red result there blocks the next release cut under the same fix-or-revert rule." and "The GKE Linux arm64 Debug leg also runs before merge when `arm_relevant=true`; a red post-merge result blocks the next release cut under the same rule." (the leg is a smoke now; the full arm64 Debug profile left GitHub CI). - "the full-profile commands in `ci-post.yml` carry no explicit `--exclude-label` arguments" and "PR CI keeps the Release smoke selected by CTest label `fast`." (the commands run in the lab's verify; GitHub CI keeps the smoke on every configuration). - "Pull requests run the Ubuntu amd64 Release smoke; post-merge runs Ubuntu amd64 and arm64 Release/Debug." and "Both architectures use the same full-profile commands." (the `build`, `build-arm` row of the job table). - docs/reference/ci-runner-routing.md: "Post-merge `ci-post.yml` runs the full profiles on x86 and arm64, plus native and parity callers; there is no macOS cell." and "the post-merge legs (`ci-post.yml`) run their full profile populations, and a documentation-only PR runs admission, preflight and the docs build while skipping proof legs." == #849 commits == --- ci: keep every CI round under 15 minutes (smoke legs in GitHub CI) Every CI round, post-merge included, now takes 15 minutes or less. GitHub CI keeps smoke legs on every configuration; the full profiles leave GitHub CI for the lab's verify, which every merge already requires on the exact head tree (pineforge/verify). Measured: post-merge run 43205609467 took 123.5 minutes (32 of 49 jobs over 15 minutes); each full-profile leg rebuilt the engine from a cold compiler cache in 11-40 minutes on a standard 4-vCPU runner, and 27 single CTest rows ran over 15 minutes on their own. PR run 43242369382 took 75.3 minutes, 73.9 of them in build-arm's unsharded ci_verify.py debug step. The parity subset step ran 13.0 and 13.75 minutes in those two runs: after the runtime it built its 54 strategy libraries one at a time (6 min 53 s), because targets named one by one on a build command line are built serially by a Makefile generator. - ci-post.yml: authz, changes, build-x86-gcc133, preflight (fast tier), smoke and smoke-arm (Release and Debug on ubuntu-24.04 and ubuntu-24.04-arm: pineforge_pr_smoke and ctest -L '^fast$'), sanitizers (the same smoke under ASan and UBSan), corpus-parity-subset and the build aggregate. Removed: preflight-slow, the shard matrices and their aggregates, the native-live call, the full corpus-parity call and the cold_native_cache input. - gke-pr.yml: build-arm runs the Debug smoke on ubuntu-24.04-arm. - corpus-parity.yml: no schedule; workflow_dispatch and workflow_call stay. - scripts/check_corpus_parity.sh --subset and CMakeLists.txt: the subset's strategy libraries build as the dependencies of one aggregate target (pineforge_corpus_parity_subset), JOBS at a time; a target the corpus does not define stops the generation. - scripts/ci_preflight.py pins the new shape; scripts/ci_release_gate.py requires the four smoke check names; the tests follow. - docs/ci.md ("Where the full suites run"), AGENTS.md and CLAUDE.md: where each cut suite now runs, the suites the lab does not run yet and the interim rule for them, and the release rule. Doc reverts, each replaced by the new shape's text in docs/ci.md: - 41f08d2: "Complete Release, Debug, native, kernel and parity profiles plus the required x86 build" (the ci-post.yml row of the trigger map). - f85ce18: "Full parity and its post-merge subset" (the corpus-parity.yml row of the trigger map). - 3a0d150: "Scheduled and manual runs execute every profile." and "Schedule, dispatch, post CI callers" (the tiers paragraph and the corpus-parity.yml row of the workflow table). - 8154993: "Dispatch and post CI callers on `main`" (the native-live.yml row of the workflow table). - 3ea5a0d: "All three build jobs refuse an OS mismatch.", "Every shard builds the full tree and runs `--ctest-shard I/N` at the runner's core count." and "Each build refuses an OS mismatch before checkout." (the post build jobs are smoke legs now; smoke and smoke-arm refuse a machine mismatch). - e7fa559: "Release uses four `build` shards, Debug two `build-debug` shards, and kernel two shards; sanitizers uses four, all on the pinned hosted images." (no post shard matrix remains). --- docs: bring the CI pages in line with the smoke-leg post-merge workflow The change that moved the full profiles out of GitHub CI left sentences in docs/ci.md and docs/reference/ci-runner-routing.md that still described the old post-merge workflow. They now say what runs: smoke legs in ci-post.yml, the full profiles and the full parity in the lab's verify, no preflight-slow job, and the measured time of the parity subset job (6.8 minutes in run 43308686886, against 13.5 and 14.3 before its strategy libraries built in parallel). docs/ci.md keeps its line count. Two comments in scripts/test_ci_preflight.py call a CI leg a leg. Doc reverts, each replaced by a sentence about the new shape: - "`preflight-slow` runs exactly those stages with `--tier slow`" and "Toolchain setup and the hash-pinned pytest install stay in the slow job." (no GitHub job runs the slow tier; the lab's preflight step runs every tier). - "the caller's `build` aggregate still requires the reusable workflow to succeed" (ci-post.yml no longer calls native-live.yml). - "retains the subset through `corpus-parity.yml` (`mode: subset`) alongside its full-parity call" (there is no full-parity call). - "The GKE Linux arm64 Debug leg runs after merge; a red result there blocks the next release cut under the same fix-or-revert rule." and "The GKE Linux arm64 Debug leg also runs before merge when `arm_relevant=true`; a red post-merge result blocks the next release cut under the same rule." (the leg is a smoke now; the full arm64 Debug profile left GitHub CI). - "the full-profile commands in `ci-post.yml` carry no explicit `--exclude-label` arguments" and "PR CI keeps the Release smoke selected by CTest label `fast`." (the commands run in the lab's verify; GitHub CI keeps the smoke on every configuration). - "Pull requests run the Ubuntu amd64 Release smoke; post-merge runs Ubuntu amd64 and arm64 Release/Debug." and "Both architectures use the same full-profile commands." (the `build`, `build-arm` row of the job table). - docs/reference/ci-runner-routing.md: "Post-merge `ci-post.yml` runs the full profiles on x86 and arm64, plus native and parity callers; there is no macOS cell." and "the post-merge legs (`ci-post.yml`) run their full profile populations, and a documentation-only PR runs admission, preflight and the docs build while skipping proof legs." == #846 commits == --- Document that hosted x86 pull-request jobs run only the fast set Say in docs/ci.md that the hosted x86 pull-request jobs build and build-debug run only the CTest rows labelled fast, and that the maintainers' full ci_verify.py run, carried by pineforge/verify, is the only full x86 test gate. No existing sentence is removed or rewritten. --- Keep docs/ci.md at its pinned line count The fast-set sentence added six physical lines to docs/ci.md, and two assertions in scripts/test_ci_preflight.py pin that page at 2133 lines. Join the sentence onto the line it follows, so the page has 2133 lines again. No word changes and no other line moves. == #845 commits == --- Give ta.pvt no value on its first bar TradingView's price-volume trend adds a term that needs the previous bar's close, so its first bar has no value; the engine returned 0 there. The first valid bar now returns na and the cumulative sum starts from zero as before, so every later value is unchanged. Invalid inputs and a zero previous close behave as before, recompute follows compute, and the other volume accumulators are not changed. --- Test that ta.pvt has no value on its first valid bar A class test with an independent forward sum on four invented bar sequences (rising, falling and flat bars; leading missing closes; missing volume; a zero previous close) checks that the first valid bar has no value, that every later value equals the forward sum, and that recompute agrees with compute, including a recompute before any compute. The two older checks that pinned the first bar to zero now expect no value. --- Note the ta.pvt first-bar change in the changelog The first bar of ta.pvt has no value, as on TradingView; it returned 0 there before. Later values are unchanged. --- List the ta.pvt changelog entry beside the other ta entry The entry moves from the first position of the Unreleased list to the position after the ta.percentile_nearest_rank entry. Its words are unchanged. Other open changes add their entries at the first position, so this one no longer meets them there. --- Give ta.obv no value on its first bar TradingView's on-balance volume adds a term that needs the previous bar's close, so its first bar has no value; the engine returned 0 there. The first valid bar now returns na and does the same state updates as before, so the running sum starts from zero and every later value is unchanged. Invalid inputs behave as before and recompute follows compute. The other volume accumulators are not changed: on their first bar they already give TradingView's value. --- Test that ta.obv has no value on its first valid bar A class test with an independent forward sum on four invented bar sequences (rising, falling and equal closes; leading missing closes; a missing volume; zero volume and a zero close) checks that the first valid bar has no value, that every later value equals the forward sum, and that recompute agrees with compute, including a recompute before any compute and recomputes with other inputs than compute saw. One such recompute row is added for ta.pvt as well. The two older checks that pinned the first bar of ta.obv to zero now expect no value. --- Note the ta.obv first-bar change in the changelog The entry for ta.pvt now names ta.obv as well: both have no value on the first bar of their series, as on TradingView, and every later value is unchanged. --- Point the private-fixture evidence at the moved test list lines The two first-bar tests add two lines to tests/CMakeLists.txt, so the line that tests/private_fixture_reads.evidence cites moved from 3473 to 3475, and the line its reason names moved from 3490 to 3492. Only those two numbers change. == #847 commits == --- test: exercise source portability guard failure boundaries Add direct-call, physical-line and fail-closed input discriminators before the portability scanner repairs. The test-only revision deliberately exposes the existing spelling and form-feed defects in the following remote RED run. --- fix: detect direct sync calls using physical source lines Recognize bare sync calls alongside global calls while excluding members. Keep reachability and reported call positions on the same newline basis. == #842 commits == --- Build an N-day chart on the year-restart N-day grid A strategy whose chart timeframe is N days (2D, 3D, 5D ...) built from daily bars ran on the days-since-1970 grid, and the chart's bar count, last_bar_index, barstate.islast and open-trade row came from the wrong bars. TradingView builds an N-day chart from N consecutive trading dates counted from the first trading date of each calendar year (the last bucket of a year is cut at its end), and its last_bar_index, barstate.islast and open-trade row follow that chart's own last bar. The engine already had that partition for request.security; this change installs it for the chart itself. The scheduler creates the existing partition source for an aggregated chart with a day count of two or more and hands it to the execution consumer, whose script bucket cursor, interval lookup, chart close and chart-time offsets read it. Only this chart path drops a leading bucket that opens before the input range and seals the final historical bucket. The calendar gains an interval overload that takes the partition's adjacent opens. The epoch-key paths, request.security, the D, W and M charts, intraday charts and runs whose input and chart timeframes are equal do not change, and a run without a partition adds nothing to the state hash. A day count whose seconds do not fit the legacy int conversion is calendar-only. Limits: a live stream keeps the previous chart path, and a selected window on a generated N-day chart partition is refused with the existing coded stop window_mode_unsupported (capability selected_window_chart_timeframe), because the window planner reconstructs the nominal calendar. Time offsets, exchange calendars and intraday input are not measured against TradingView. tests/test_n_day_chart_timeframe.cpp holds the rows on synthetic daily bars: the 3D grid, the last bar index, barstate.islast, the open row at the end, the 2D and 5D grids, the stream and refusal rows, the time offsets and the controls that must not move. It is registered through tests/cr150.d/n-day-chart-timeframe.cmake. scripts/check_native_cpp_versions.py lists the new reads of the retained input bars in the scheduler. TradingView tapes (sha256): 3D chart 1b2ef41a6f1ba4096e6659e12d5ef3f03c8bf2a51e4bb0d8331d814d9aade262, 2D chart ea679d91cbc0cd078c80604e2c73b3c9bb3104ff8d1989695da6abecb8b9e174, 5D chart 5342be10bdda8185b12b2c6a95b9b26101334cd1f58663ceb1f28c42197cb5dc, last_bar_index 05617c83d33cbd8be7a0d64356dc8f40e5b8d2ab20baa1b74b08b8aa66a07d25, barstate.islast 4ae8e0a932b7da6ffb4e0bcd5de9ee4d514cd6b78dbbb3fa4207aba1a2c6420d. Refs #811. --- Re-anchor the line citations of the published pages (digits only) The N-day chart commit moved code in src/native_execution_consumer.cpp, src/native_execution_consumer.hpp, src/source/pine_strategy_host.cpp, src/source/pine_scheduler_native.cpp and src/timeframe.cpp, so the file:line citations of the published pages that point into them went stale, and scripts/check_doc_anchors.py reported 212 bad anchors. This commit changes digits only: the line numbers inside file:line citations of nine pages (145 changed lines, 233 changed numbers) and nothing else. No sentence, word, content pin or table is edited. The numbers were moved with the repository's own tools: scripts/check_doc_anchors.py --fix (the cited symbol sits on exactly one line) and scripts/reanchor_doc_citations.py --ref 360c4ab (a line map from the commit before the N-day chart change, where the pages were correct). The last six citations, which claim no symbol, were carried across the same line map by hand: the cited code line is the same text at its new number. scripts/check_doc_anchors.py reports no bad anchor on the result, and scripts/check_doc_reverts.py reports nothing, since a re-anchor is not an edit. --- Floor an explicit close quantity to the quantity step strategy.close(id, qty = X) with an X that is not a multiple of the declared quantity step (qty_step, or mincontract when no qty_step is declared) placed no closing order: the quantity reached the native order layer unchanged, and that layer refuses explicit units that are off the grid. TradingView floors such a quantity to the step and closes that much; a quantity that floors to zero, or is larger than the position, closes the whole position. An explicit close quantity now goes through the same floor as an explicit entry quantity, including its handling of a quantity a hair under a step multiple, before the order is made. A quantity that floors to zero closes the whole exposure of the id when there is any. An explicit strategy.exit quantity is floored the same way before its units are reserved; one below a single step is ignored, so the exit covers the whole position (with the default qty_percent), as TradingView does. Runs that declare no quantity step are unchanged. The new test runs the adapter on synthetic bars with a step of 0.00001 and pins the quantities TradingView closed on BTCUSDT 1-hour bars: a half close of 0.03683 takes 0.01841 and leaves 0.01842, a close of 0.000004 or 0.123456 takes the whole position, a limit exit of 0.000004 or 0.123456 takes the whole position at its limit, and a limit exit of 0.018415, a short side and a reducing order floor the same way. One more row checks that a close quantity a hair under a step multiple (0.018409999999999) closes 0.0184, the same as an entry of that quantity opens, and leaves 0.01843. The lines added to src/source/pine_adapter.cpp move the lines the documentation cites below them; the anchors that follow them are re-pointed by the number of lines added (digits only, no sentence changes). Refs #716. --- Re-anchor the line citations of the published pages (digits only) Merging main moved code in src/native_execution_consumer.cpp, src/source/pine_strategy_host.cpp, src/source/pine_scheduler_native.cpp and the other files the N-day chart change touches, so the file:line citations of the published pages that point into them went stale, and scripts/check_doc_anchors.py reported 208 bad anchors on the merge. This commit changes digits only: the line numbers inside file:line citations of nine pages (140 changed lines) and nothing else. No sentence, word, content pin or table is edited. The numbers were moved with the repository's own tools: scripts/check_doc_anchors.py --fix (the cited symbol sits on exactly one line) and scripts/reanchor_doc_citations.py --ref e163980 (a line map from main, where the pages are correct). The last six citations, which claim no symbol, were carried across by hand: the cited code line is the same text at its new number (one of them, the cursor constructor, gained a parameter on its line). --- Fail the run when a timezone lookup cannot be carried out The check that accepts a timezone name written in a script looks the name up in the timezone database with realpath, open and read, and it treated every failure of those calls as "no such zone". When the process had no file descriptor or memory left, or a read failed, a valid name such as America/New_York was refused with pine_timezone_error (class strategy, not retryable) and TradingView's text for an incorrect timezone; a database directory that could not be resolved refused every zone name the same way. The lookup now tells an absent name from a lookup that could not be carried out. ENOENT, ENOTDIR, ELOOP, ENAMETOOLONG and EISDIR still mean that the name denotes no zone file and keep pine_timezone_error. ENOMEM raises out_of_memory. Every other error raises engine_unclassified_error with the errno kept and the failed call named in the message. When the database directory itself cannot be resolved, a missing directory included, the run fails the same way and the message names the directory. In this check an interrupted call is tried up to three times, and a header read that returns fewer than four bytes is continued. Only the argument check of script timezones raises, through the internal timezone_lookup_accepted; the installed native_calendar::timezone_accepted keeps its false result, so its callers, the selected-run admission among them, refuse as before. A name that is also a complete POSIX definition, such as EST5EDT, stays accepted when its optional file lookup fails. A failed lookup is not remembered as an accepted name. tests/test_timezone_lookup_failure.cpp interposes realpath, open and read to fail each call of the lookup with each class of error, and checks the failure code, the errno and the call named in the message, that the name is looked up again afterwards and a clean run then books the same trades, that one interrupted call does not change the run, the file header (wrong magic, end of file, a read split in two), the database directory, the selected-run admission, and a TZDIR that is not a directory, where UTC offsets and POSIX definitions still run. Documentation sentences this change removes or rewrites: - docs/coverage.md, the pine_str_format_time row: "The message is TradingView's text with the engine's bar prefix, `Error on bar N: `, or `Error before the first bar: ` when no bar is being executed, and the arguments carry `bar_index`." It now begins "The error message for a refused name is", after a new sentence on the lookup failures. The allocation-bound manifest lists one new group, the message string of the lookup refusal in the native calendar (ceiling 1418 -> 1419, pinned count 1356 + 63 + 184), and its site inventory is re-emitted. --- Test the void setter record under consecutive allocation failures A failed void C setter records its failure in the handle's run consumer. If a handle had no consumer, recording would first allocate one, and a failure of that allocation would leave only the status, without the code or the text. That state cannot be reached: the engine's only constructor binds the consumer, and the native host can be neither copied nor moved. These tests pin that, and show that the record survives a second allocation failure, without a change to the engine. - Every host type of the test starts with its consumer: storing a selected-window descriptor allocates nothing and does not return the allocation refusal -4. - When a setter's own string and then the rendering of the record's bar prefix are both refused, the setter's out_of_memory record is read without any allocation and its text falls back to "setter failed"; the next run start fails once with that record and publishes no report, and the start after it produces the report of a clean run. A setter refusal that carries its own code keeps that code when the rendering fails. - A sweep refuses two, and then three, consecutive allocations starting at every allocation index of a configured run (the setters, preparation, the bar callbacks and the report). Each injected run ends in a typed setup refusal before the first bar, in out_of_memory after it, or in a report byte-identical to the clean run, and the sweep prints how many refusals each run received: a pair at 256 indexes (Debug 258), a triple at 37 (Debug 39); the other 234 end at the first refusal. tests/fail_alloc_seam.hpp gains arm_span(first, count), which refuses count consecutive allocation attempts, and refusals(), the number of refusals delivered; arm(nth) is arm_span(nth, 1). --- Name the timeframes and the rule when a run's timeframes do not fit A run whose input, script or intrabar timeframes do not fit was refused with "Pine adapter produced invalid native run spec field 29" (or 4), and an args object that carried only the option name. The text read like an internal failure, the number was an internal enum value, and neither named the timeframes or the pairing that failed. The refusal now says which timeframes are involved, why they do not fit and what to pass instead, for example "input timeframe '2' cannot build script timeframe '15' (15 is not a multiple of 2); pass bars whose timeframe divides the script timeframe '15', for example 1-minute bars." The args keep the option name and add input_tf, script_tf, intrabar_tf when the bar magnifier is on, and a reason: not_divisible, script_finer_than_input, intrabar_not_divisible or script_finer_than_intrabar. The code stays run_options_rejected, which runs are refused or accepted does not change, and every other refusal keeps its text and args. The catalog declares the two new optional arguments and the new message templates; the generated registry, the catalog diff and the failure-code tables are regenerated by their scripts. The bar magnifier page says that passing the intrabar timeframe itself as the input is refused when it does not divide the chart timeframe. The allocation-bound manifest lists the refusal helper's three new groups (its failure arguments, rule text and message text; ceiling 1418 -> 1421, pinned count 1602 -> 1605) and its site inventory is re-emitted. The lines added to src/source/pine_adapter.cpp move the lines the documentation cites below them; the anchors that follow them are re-pointed by the number of lines added (digits only, no sentence changes). Refs #813. --- Record a warning for a request.security timeframe that is not a multiple of the feed's A request.security call whose requested timeframe is not a multiple of the timeframe of the feed that evaluates it (75 minutes on a 60-minute feed) is accepted, and the script then reads the feed's own bars instead of the requested timeframe's. The run now records one named warning for each such site, request_timeframe_not_multiple, with the feed and requested timeframes and the site identity, and exposes the warnings through the new strategy_get_run_warnings accessor as one canonical JSON array. The accessor is additive: no struct is added and no existing symbol or layout changes (ABI 5, stream layout 4). It answers "[]" when the run recorded no warning, before any run, and after a later attempt that was refused before it began; a live stream's warnings stay readable. If collecting the warnings fails it answers a fixed run_warnings_unavailable element, never an empty list. Values, trades, run acceptance, the failure accessors and the state hashes are unchanged. Compatible pairings, lower-timeframe-array sites and sites with no loaded bars stay silent. The two canonical-JSON appenders of the run-failure writer move unchanged into a private header, so the warnings use the same writer. The Docker report and scripts/run_strategy.py read the optional getter after the report is assembled and before the strategy handle is released, and show a nonempty list as diagnostics.warnings. A library without the getter, or a run without warnings, produces the same report bytes as before. Unknown warning codes and their arguments are kept, malformed elements are dropped, and the warnings stay out of sink records and fingerprints. scripts/run_strategy.py also returns the list, prints one summary line and includes it in a requested trace JSON. The new tests run both entry points against a copy of the stdout stub that adds a canned warning, a fallback warning and selected-window support, and against a build of the same stub without the getter. The allocation-bound manifest lists the ten new groups of the warning records and their strings (ceiling 1421 -> 1431, pinned count 1605 -> 1615), points the entry of the moved appenders at the private header, and its site inventory is re-emitted. The lines added by this change move the lines the documentation cites below them; the anchors that follow them are re-pointed by the number of lines added (digits only, no sentence changes). Refs #718. --- Keep the run warnings getter read-only on the current C boundary The void C entry point wrapper now takes the strategy handle and records an escaped exception on it as the handle's failure. The run warnings getter was written against the earlier one-argument wrapper and no longer compiled. It now passes no handle, so it still records nothing on the handle, as before; the function it calls cannot throw. Co-authored-by: Claude Opus 5.5 <[email protected]>
Every CI round of this repository, post-merge included, now takes 15 minutes or less. GitHub CI keeps fast smoke legs on every configuration; the full profiles leave GitHub CI for the lab's verify, which every merge already requires on the exact head tree (
pineforge/verify).Why
CI post-mergeon main): 49 jobs, 123.5 minutes end to end, 32 jobs over 15 minutes. The critical path wasnative-live (live-sanitizers)(122.9 min) andnative-live (live-tsan)(120.2 min), each one unsharded verify step.GKE PR): 75.3 minutes;build-armalone ran 74.5 minutes, one unshardedci_verify.py debugstep of 73.9 minutes. The three code PRs before it took 75.4, 74.7 and 78.6 minutes.report_default_delta_work65.1 min (live-sanitizers) and 44.7 (live-tsan);test_retirement_work_scaling45.5 (sanitizers);delivery_pending_tail_work44.1 (live-tsan) and 35.0 (live-sanitizers);test_ci_verify_slice_1to_616.1-31.7 (kernel-only, native, Release);test_cr559_map_capacity_cli26.7-30.8 (Release, arm64 Release, native);test_cr559_map_capacity_calc_on_fill26.3 (sanitizers);test_projected_pending_scaling17.8 (sanitizers);test_cache_policy_v115.6 (sanitizers); fournative_live_equivalence_e2e_*rows 15.3-18.3 (live-sanitizers).What GitHub CI runs now
ci-post.yml(push to main, the 01:30 UTC schedule, manual dispatch):authz,changes,build-x86-gcc133,preflight(fast tier),smoke(build (ubuntu-24.04, Release)andbuild (ubuntu-24.04, Debug)),smoke-arm(build (ubuntu-24.04-arm, Release)andbuild (ubuntu-24.04-arm, Debug)),sanitizers(the same smoke under ASan and UBSan with the full profile's build flags and run options),corpus-parity-subset, and thebuildaggregate. Each smoke leg buildspineforge_pr_smokeand runsctest -L '^fast$' --no-tests=error, cold, with no Actions cache.gke-pr.yml:build-armruns the same Debug smoke onubuntu-24.04-arm(whenarm_relevant); every other job is unchanged.corpus-parity.yml: no schedule;workflow_dispatchandworkflow_call(the post-merge subset) remain.The parity subset leg
Check TradingView parity over the pull-request subsetran 12 min 59 s in post-merge run 43205609467 and 13 min 45 s in PR run 43242369382. It is the longest leg both workflows keep, so both would have ended between 14 and 15 minutes with no margin.-jsays: in run 43205609467 the runtime was built after 2 min 12 s and the 54 libraries then took 6 min 53 s, one at a time.scripts/check_corpus_parity.sh --subsetnow passes the same target names to CMake as one list (PINEFORGE_CORPUS_PARITY_SUBSET_TARGETS) and builds one aggregate target,pineforge_corpus_parity_subset(top-levelCMakeLists.txt), whose dependencies buildJOBSat a time. The aggregate is outside the default build, and a target the corpus does not define stops the generation. The lab'ssubsetstep runs the same script.baseline match=54, moved=0):TradingView parity holds on the 54-probe subset. A target the corpus does not define was refused by name at generation. The hosted-runner time of the leg is in the run of the Proof section.Where every cut leg and row now runs
build-shard (ubuntu-24.04, Release, 1-4/4)and its aggregatereleasesteptest_ci_verify_slice_*,test_cr559_map_capacity_clibuild-shard (ubuntu-24.04, Debug, 1-2/2)and its aggregatedebugstepsanitizers-shard (1-8/8)and its aggregatesanitizerssteptest_retirement_work_scaling,test_cr559_map_capacity_calc_on_fill,test_projected_pending_scaling,test_cache_policy_v1kernel-only-shard (1-2/2)and its aggregatekernelsteptest_ci_verify_slice_*preflight-slowpreflightstep (tierallkeeps every slow stage)corpus-parity(full) and thecorpus-parity.ymlscheduleparitystep (the samescripts/check_corpus_parity.sh, full mode)build-shard (ubuntu-24.04-arm, Release, 1-4/4)test_cr559_map_capacity_clibuild-shard (ubuntu-24.04-arm, Debug, 1-4/4)and the PR's full arm64 Debug verifynative-livecall:native(4 shards)test_ci_verify_slice_*,test_cr559_map_capacity_clinative-livecall:live-sanitizers,live-tsanreport_default_delta_work,delivery_pending_tail_work,native_live_equivalence_e2e_*src/orinclude/still gets the full set; a workflow-only change getspreflightanddocs; a test-only change gets the build profiles without the parity steps.src/orinclude/and before any release tag.ci-post.ymlrun (the smoke legs) on its commit and the lab's verify evidence for its tree.scripts/ci_release_gate.pynow requires the four smoke check names; it already requiredpineforge/verifyandpineforge/parity.native-live.ymlstays as a manual-dispatch workflow (its profiles are the recipe for the lab steps).Contract and tests
scripts/ci_preflight.pypins the new shape: the post-merge job set, events and gate step; the smoke strategy and check names; apost_smoke_workflow_findingscheck (no Actions cache, each smoke job's exact step list in order, the machine guard, the install line, the exact smoke command, the sanitizer flags and run options); the PR arm smoke;corpus-parity.ymlwithout a schedule.scripts/test_ci_release_gate.pyandscripts/test_ci_preflight.pyfollow the new shape; tests of removed structures are replaced by tests of the new ones (dropped build type or job, renamed check, non-hosted runner, container, cache, added step or package, changed smoke command, continue-on-error, gate results).scripts/test_ci_verify.py:ci-post.ymlleaves the list of places that must point at the profile floor table (it runs no full profile); the same test still refuses a hard-coded floor there.docs/ci.md("Where the full suites run"),AGENTS.mdandCLAUDE.mddescribe the new shape.The
GKE PRcheck on this pull requestIt is red at
admit, and that is expected. On pull requestsgke-pr.ymlruns main's copy of the rules (scripts/ci_preflight.py --rules-only) over the head's workflow files, and main's rules pin the job set this change replaces: 36 findings, every one a pin of the removed shape (for example "ci-post.yml is missing job preflight-slow", "gke-pr.yml build-arm must run the complete Debug profile", "ci-post.yml job smoke-arm must run on ubuntu-24.04 for post-merge events"). The head's own rules accept the head's files. After the merge, an open pull request must merge main before itsGKE PRcheck passes again.Doc reverts
Each is replaced by the new shape's text in
docs/ci.md:ci-post.ymlrow of the trigger map).corpus-parity.ymlrow of the trigger map).corpus-parity.ymlrow of the workflow table).main" (thenative-live.ymlrow of the workflow table).--ctest-shard I/Nat the runner's core count." and "Each build refuses an OS mismatch before checkout." (the post build jobs are smoke legs now;smokeandsmoke-armrefuse a machine mismatch).buildshards, Debug twobuild-debugshards, and kernel two shards; sanitizers uses four, all on the pinned hosted images." (no post shard matrix remains).A squash merge must keep this section in the squash message: the doc-revert check on
mainreads it there.Proof
PENDING: one dispatched
ci-post.ymlrun on this head (workflow_dispatchon the branch), every job's wall time listed here.gke-pr.ymlruns main's copy of the workflow on pull requests (pull_request_target), so its newbuild-armis first measured on the first pull request after this merge.🤖 Generated with Claude Code