Skip to content

Commit 669de8d

Browse files
committed
Merge current main into release archive compatibility changes
2 parents b97140d + 09756a9 commit 669de8d

8 files changed

Lines changed: 320 additions & 267 deletions

File tree

‎AGENTS.md‎

Lines changed: 110 additions & 143 deletions
Large diffs are not rendered by default.

‎CLAUDE.md‎

Lines changed: 107 additions & 123 deletions
Large diffs are not rendered by default.

‎docs/design/native-feature-parity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -365,7 +365,7 @@ the campaign's own anchor audit found that the first thing to rot:
365365
366366
### 3.1 Parity discipline for every lane (R5-10) — stated once, referenced per lane
367367
368-
- **(a) NEUTRALITY.** Every new behaviour is opt-in by a spec field or a new request kind. The adapter's `project()` (pine_adapter.cpp:2916-3197) never sets the field and never emits the kind, so adapter runs are byte-identical by construction. The four feed-shape fields are folded into `hash_spec` unconditionally today, including defaults (native_execution_consumer.cpp:252-360), so even a defaulted new field would move every continuation hash; the precedent for a conditional digest is `precommit_digest_` (native_execution_consumer.hpp:2131). Each lane adds a test that a default spec hashes to the pre-change constant. Every new durable field is hashed (`check_broker_state_hash_coverage.py`, fail-closed, waivers in `broker_state_hash_waivers.txt`). Proof per PR: `scripts/run_corpus.sh` + `scripts/verify_corpus.py` locally, then ONE campaign sweep on the final tree, run by the maintainers (fixed population, 4190 probes when this plan was written, zero tolerance on the hard band — native-refactor-progress.md:42-48; a fresh exact-HEAD `PASS` verdict is required, `sha256:59adeba8aabaefa740633d52e9216b49065b28b397b87f22c57ad5e19f2948c2` AGENTS.md:149-150). A lane that cannot be shown neutral by construction does not merge.
368+
- **(a) NEUTRALITY.** Every new behaviour is opt-in by a spec field or a new request kind. The adapter's `project()` (pine_adapter.cpp:2916-3197) never sets the field and never emits the kind, so adapter runs are byte-identical by construction. The four feed-shape fields are folded into `hash_spec` unconditionally today, including defaults (native_execution_consumer.cpp:252-360), so even a defaulted new field would move every continuation hash; the precedent for a conditional digest is `precommit_digest_` (native_execution_consumer.hpp:2131). Each lane adds a test that a default spec hashes to the pre-change constant. Every new durable field is hashed (`check_broker_state_hash_coverage.py`, fail-closed, waivers in `broker_state_hash_waivers.txt`). Proof per PR: `scripts/run_corpus.sh` + `scripts/verify_corpus.py` locally, then ONE campaign sweep on the final tree, run by the maintainers (fixed population, 4190 probes when this plan was written, zero tolerance on the hard band — native-refactor-progress.md:42-48; a fresh exact-HEAD `PASS` verdict is required, `sha256:59adeba8aabaefa740633d52e9216b49065b28b397b87f22c57ad5e19f2948c2` AGENTS.md:50-51). A lane that cannot be shown neutral by construction does not merge.
369369
- **(b) TWIN.** One chosen probe per lane runs through the adapter AND a hand-written `NativeStrategyHost` on the same bars and spec. Trade lists are diffed **by identity** (entry / exit time, price, qty, pnl, commission), never by trade number; then event, ownership and hash detail (S's D → E order). A difference is allowed only where a named TV-quirk row of §1 is active, and each one is itemized. Because 430 test files drove the adapter and 14 the native host when this plan was written (O; the line snapshot has 486 `tests/*.cpp` naming `source::PineStrategyHost` and 128 naming `NativeStrategyHost`, while the main snapshot has 543 and 110 respectively), every lane also adds native-only tests: a lane is not accepted on the corpus alone, nor on a unit test alone (S).
370370
- **(c) EPOCH BUDGET.** Three inline-namespace epochs move — the request epoch, the host epoch and (*fresh*, not in the inputs) the run-spec epoch, which every new spec field moves. They stood at `native_order_v5`, `engine_script_run_v17` and `native_run_spec_v2` when this plan was written; on this tree they are `native_order_v8` (native_order.hpp:45), `engine_script_run_v19` (native_host.hpp:20) and `native_run_spec_v3` (native_run_spec.hpp:15): the planned batch spent the budget below once; the later host value epoch advanced to v19, and 1.8.0 advances the order values to v8 for the changed layouts. One bump each per release batch: `native_order_v6` = L3 + L7 + L4's origin / event; `engine_script_run_v18` = L4's host hooks + L5 + L6 virtuals + L2's `hash_host_extension` + §2.ii a, f; `native_run_spec_v3` = the spec blocks of L2, L4, L5, L6, L8 (L9 if it ships in the batch). A lane that lands before its batch closes stages behind the open epoch, so codegen-built strategies rebuild once per batch. Epoch lanes are not neutral-refactor lanes: the implementation plan lists the header extension explicitly; the frozen C++ ABI fixtures (`tests/fixtures/native_cpp_abi/host-*`) and the ABI checkers (`check_native_cpp_abi.py`, `check_settlement_cpp_abi.py`) move with them; `variant_size` pins change (`OrderIntent` native_order.hpp:250, `CommandEvent` native_order.hpp:1400). Every new test unit compiles first against the frozen previous-epoch header closure (fail-before, `CLAUDE.md`). <!-- verified HEAD -->
371371
- **Worker verification** (`CLAUDE.md`): `ci_preflight.py`, then `ci_verify.py release`; plus `check_c_abi_runtime.py` and `check_native_include_independence.py`. No campaign sweep from a worker.

‎docs/reference/README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Reference text moved out of CLAUDE.md and AGENTS.md
2+
3+
`CLAUDE.md` and `AGENTS.md` are short on purpose: how to build, how to test, and the rules that matter. The longer text that is not testing guidance lives here. Each file below is the exact text of the line range shown, copied unchanged (no header added) from the two files as they stood at commit `4c9e7633`.
4+
5+
| File | Moved from | What it holds |
6+
| --- | --- | --- |
7+
| [`c-abi-export-sop.md`](c-abi-export-sop.md) | `CLAUDE.md` lines 76-99 and `AGENTS.md` lines 73-96 (identical) | checklist for adding or removing a runtime `PF_API` export (the `source-guard-c-abi` gate) |
8+
| [`refactor-workers.md`](refactor-workers.md) | `CLAUDE.md` lines 108-143 and `AGENTS.md` lines 105-140 (identical) | rules for an agent implementing a slice of the native-engine refactor |
9+
| [`ci-runner-routing.md`](ci-runner-routing.md) | `AGENTS.md` lines 9-19 | merge ruleset summary and CI runner routing as written then |
10+
| [`merge-gate-and-promotion.md`](merge-gate-and-promotion.md) | `AGENTS.md` lines 142-160 | merge gate, parity campaign and baseline promotion as written then |
11+
12+
These files are a record, not a second source of truth. Where one disagrees with `CLAUDE.md`, the workflow files under `.github/workflows/` or [`docs/ci.md`](../ci.md), those win. For example `ci-runner-routing.md` still describes the PR build as `ci.yml` and a macOS post-merge cell, while the pull-request proof jobs now live in `gke-pr.yml` and `ci-post.yml` has no macOS cell.

‎docs/reference/c-abi-export-sop.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
## SOP: adding a runtime `PF_API` export (CI gate — recurring failure)
2+
3+
CI runs `python3 scripts/check_c_abi_runtime.py` as the `source-guard-c-abi`
4+
stage of `scripts/ci_preflight.py` and of every `scripts/ci_verify.py` profile,
5+
before anything is built. It pins the exact set of `PF_API` symbols implemented
6+
in `src/c_abi.cpp` against the hardcoded `EXPECTED_RUNTIME` frozenset in that
7+
script. Adding (or removing) a runtime export WITHOUT updating that list fails
8+
the `source-guard-c-abi` stage, and with it preflight and every profile, even
9+
though the build and ctest would be green.
10+
11+
Checklist when touching runtime exports — update ALL of these together:
12+
13+
1. `src/c_abi.cpp` — the implementation (and its file-header symbol comment).
14+
2. `include/pineforge/pineforge.h` — the `PF_API` declaration (+ doxygen).
15+
3. `scripts/check_c_abi_runtime.py` — add the symbol to `EXPECTED_RUNTIME` and
16+
bump `EXPECTED_PUBLIC_DECLARATIONS` / `EXPECTED_RUNTIME_IMPLEMENTATIONS`.
17+
4. Python ctypes harnesses if consumers must call it
18+
(`scripts/run_strategy.py`, `tutorial/run*.py`, `docker/run_json.py`,
19+
`benchmarks/throughput/grid_search_repro.py`).
20+
5. README symbol table if it enumerates exports.
21+
22+
Before pushing: `python3 scripts/check_c_abi_runtime.py` (must exit 0).
23+
Per-strategy symbols (strategy_create, run_backtest, …) are NOT in this list —
24+
they are codegen-emitted; the checker enforces exactly that split.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
The merge ruleset requires `pineforge/verify` and `pineforge/parity` commit
2+
statuses on the PR head's exact tree. The maintainers post them from their own (private)
3+
verification tooling after full verification on their x86-64 build hosts and a
4+
parity verdict; an outside contributor cannot post them. GitHub Actions is advisory: PRs build Ubuntu Release only (`ci.yml`); Debug, sanitizers, kernel-only and the macOS cell run post-merge (`ci-post.yml`).
5+
The PR build (`ci.yml`) runs only the `fast`-labelled CTest rows (`ctest -L '^fast$'`); the post-merge legs (`ci-post.yml`) run
6+
every row, and a documentation-only change runs preflight alone among verification jobs.
7+
The seven PR/post-merge verification and documentation workflows start with the canon `authz` job and every other job in them needs it; release uses its status gate and baseline promotion its `promote-guard`. Pull-request
8+
proof workflows and secret-free post jobs use exact `gke-*` pins in runner group 14. Credentialed release, notification, promotion and probe jobs use `gkp-2cpu-4gb-amd64`; docs deployment uses `gkp-4cpu-8gb-amd64`, all in protected runner group 15. Hosted PR light jobs and `docs.yml`'s `build` retain `ubuntu-24.04`; post guards and aggregates use their exact light GKE pins.
9+
Hosted fork PRs receive no secrets, so they need no VM-only fork-refusal hook. Retired main/post VM labels remain refused. Group 14's mirror is `.github/runner-group-allowlist.txt`; group 15's required refs are in `.github/runner-group-15-allowlist.txt`, with live membership confirmed by the supervisor.
10+
Release's x86_64 prebuilt cell uses `gkp-16cpu-32gb-amd64-128gb`, installs Eigen and asserts exactly 3.4.0 before CMake; the arm64 cell retains hosted `ubuntu-22.04-arm`. App-token jobs keep their exact `release`/`release-bot` environments. The `build-gate` job in `ci-post.yml` and `native-live.yml`'s `native-live-gate` always run and refuse a non-success authz result first.
11+
See `docs/ci.md` for the class table and docs-only exception. The commands below run the full sets.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
## Merge gate and parity campaign
2+
3+
The `PineForge strict CI base` ruleset requires `pineforge/verify` (full
4+
`ci_verify.py` profiles on the PR head's exact tree) and `pineforge/parity`
5+
(no parity regression, or no engine behaviour change). The maintainers'
6+
private verification tooling posts both commit statuses; GitHub Actions does not post them.
7+
Its PR jobs provide faster advisory feedback and a documentation-only change runs preflight alone among verification jobs; otherwise push-to-main and manual
8+
dispatch run the full CI profiles. A campaign PASS verdict still binds the
9+
exact engine and codegen HEADs for baseline promotion.
10+
11+
Merged single-axis PRs advance the campaign baseline automatically through
12+
`.github/workflows/promote-baseline.yml`, installed from the private
13+
pineforge-workflow repository's `campaign/ci/promote-baseline.yml`. `main`'s own copy runs
14+
on a `push` to `main` (it resolves the merged PR from the pushed commit) and runs no PR code. It promotes only
15+
when the merge commit on `main` carries the gated PR head's tree (PRs are
16+
squash-merged, so the head itself never lands on `main`), both required
17+
statuses are success on that head, and the maintainers' private campaign tool finds the merge gate's
18+
parity verdict for that tree; it then adopts the pair that verdict measured.
19+
Anything else exits green without promoting.

‎docs/reference/refactor-workers.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
## Refactor workers (native-engine programme, R4 and later)
2+
3+
Applies to any agent — Claude, Codex, OpenCode — implementing a slice of the
4+
native-engine refactor in this repository. The maintainers' campaign
5+
repository (`pineforge-workflow`, private; its `AGENTS.md`, "Roles and dispatch")
6+
governs who dispatches, reviews and measures; this section is what binds you here.
7+
8+
- You work in the worktree and branch your brief names, on the files it
9+
lists as yours, and nowhere else. A need in another file is reported to the
10+
supervisor, never edited quietly. You never push; the supervisor opens the
11+
PR after the one campaign sweep.
12+
- A neutral refactor never opens the measured sources
13+
(`src/native_execution_consumer.cpp`, `src/native_order.cpp`,
14+
`src/engine_execution.cpp`, `src/engine_orders.cpp`, `src/engine_run.cpp`,
15+
`src/engine_path_resolve.cpp`, and the source-layer `src/source/*` units) and
16+
never edits the frozen native headers the settlement ABI checker names; the
17+
fixed parity population must stay byte-identical by construction, which the
18+
supervisor proves with the sweep, not you.
19+
- C++17 only (`rg 'bit_cast|<bit>' src include tests scripts` = 0). Every
20+
added durable state is hashed. Existing numeric assertions and fixture
21+
outputs stay unchanged unless the contract lists the extension.
22+
- Every new test unit is compiled first against the frozen previous-epoch
23+
header closure and its first diagnostic is recorded (fail-before). A test
24+
whose body is a seed row only, a `return 0` main, a disabled row or a TODO
25+
placeholder is a blocker to report, never evidence.
26+
- Verification is local: `python3 scripts/ci_preflight.py`, then
27+
`python3 scripts/ci_verify.py release --build-dir <your dir> --jobs N` (all
28+
suites, the settlement ABI matrix, the checker self-tests); debug,
29+
sanitizers and native profiles when the brief asks. No campaign sweep from
30+
a worker.
31+
- Your handoff is a report: HEAD sha, `git diff --stat <base>..HEAD`, the
32+
verification summary lines and log paths, every deviation from the
33+
contract with its reason, and every open question. The supervisor judges
34+
the slice by its executed result: no regression in the campaign sweep, the
35+
engine closer to an independent backtest + forward-execution state
36+
machine, and Pine-parity behaviour kept in codegen with this kernel clean.

0 commit comments

Comments
 (0)