Skip to content

Commit 69467f7

Browse files
luisleo526claude
andcommitted
CR-150 O(1) report integration: the engine 1.7.0 line as one commit over main 9eff925
One squash-equivalent commit: parent = engine main 9eff925 (#463), tree = the CR-150 picks line merged with that main (docs/ci.md the only conflict: the line's clarification paragraph and main's two baseline-promotion paragraphs are adjacent additions, both kept). The squash body at the merge carries this message verbatim. The 40 documentation sentences below were deleted by commits of the line and are intended: they are the superseded wording of the pages the final 1.7.0 text replaces. scripts/check_doc_reverts.py reads this message for them. Format per line: file | introducer short sha | start of the deleted sentence. Introducers: dbf56ed (docs/design/arena-bound.md 22), 03b2406 (docs/ci.md 7), c7e6bcb (docs/pages/scratch-tables.md 5), 1fa26e7 (scratch-tables.md 3), b4a3f76 (scratch-tables.md 1), 00dec17 (docs/scratch-store.md 1), 2688775 (docs/pages/abi-stability.md 1). docs/pages/abi-stability.md | 2688775 | "The engine validates/attaches the inherited fixed-fd-5 arena or returns the handle's typed `report_sink_failed`/`arena_backing`." docs/pages/scratch-tables.md | 1fa26e7 | 'A jailed arena must be a sealed memfd with at least `F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL`, and the jailed admission (`strategy_require_report_sc' docs/pages/scratch-tables.md | 1fa26e7 | 'Where sealed memfds do not exist (not Linux) the admission accepts the arena the host supplies.' docs/pages/scratch-tables.md | 1fa26e7 | 'With `PF_SCRATCH_ARENA_FD` set, only exactly `"1"` enables it; unset or exactly `"0"` preserves the unjailed backing rules, and every other value (inc' docs/pages/scratch-tables.md | b4a3f76 | 'A jailed arena must be a sealed memfd with at least `F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL`; ordinary files, sealing-disabled memfds and unsupport' docs/pages/scratch-tables.md | c7e6bcb | 'The engine never changes process signal disposition.' docs/scratch-store.md | 00dec17 | 'Below it the drain is held to 4,096 B for every population up to 8,191, where the resident journal events own the definitions the spilled book rows ne' docs/design/arena-bound.md | dbf56ed | 'L18 sibling footer evidence described below; absent from this base' docs/design/arena-bound.md | dbf56ed | '`erased_minima_` `include/pineforge/source/pine_adapter.hpp:N`;' docs/design/arena-bound.md | dbf56ed | '`remember_erased` `src/source/pine_adapter.cpp:N`' docs/design/arena-bound.md | dbf56ed | '`consumed_partial_exit_cycles_` `include/pineforge/source/pine_adapter.hpp:N`;' docs/design/arena-bound.md | dbf56ed | '`src/source/pine_adapter.cpp:N`' docs/design/arena-bound.md | dbf56ed | 'The exact external file:line evidence is recorded in the lane report; these sources must be merged and their local anchors refreshed before claiming c' docs/design/arena-bound.md | dbf56ed | 'Summing all eleven peaks gives:' docs/design/arena-bound.md | dbf56ed | 'There are eleven persistent regions and up to three extra replacement indexes.' docs/design/arena-bound.md | dbf56ed | 'Each independently rounded region wastes less than C.' docs/design/arena-bound.md | dbf56ed | 'Summing independent peaks and adding fourteen complete chunks is conservative even when their real peak times differ.' docs/design/arena-bound.md | dbf56ed | 'This covers chunk fragmentation, not unlisted stores or a missing population proof.' docs/design/arena-bound.md | dbf56ed | 'At L=64 the position term is **94*T**; each placement data term is **104*D+168*Q**; consumed-cycle data is **169*Q**; the root term is **16*R**.' docs/design/arena-bound.md | dbf56ed | "The contract's eleven coefficients agree with the inspected code and L19l-A/C reports." docs/design/arena-bound.md | dbf56ed | 'No numeric correction to those coefficients was needed. “72..104 bytes/key” in the L19l-C steady-state explanation does not include the rehash overlap' docs/design/arena-bound.md | dbf56ed | 'Each of two placement data files' docs/design/arena-bound.md | dbf56ed | 'Each of two placement index peaks' docs/design/arena-bound.md | dbf56ed | 'Eleven contributions in the table above, including both two-file rows.' docs/design/arena-bound.md | dbf56ed | 'D=32,Q=1,L=1,E=1; both 128-slot rehash peaks are 3072;' docs/design/arena-bound.md | dbf56ed | 'No production header declaration is changed.' docs/design/arena-bound.md | dbf56ed | 'Linux linker wrappers reject write/pwrite/ftruncate on unlinked regular scratch files, while allowing the named report sink.' docs/design/arena-bound.md | dbf56ed | 'The current base still has those owning-port dependencies.' docs/design/arena-bound.md | dbf56ed | 'It must not be reported as corpus proof until they land.' docs/pages/scratch-tables.md | c7e6bcb | 'The report sink stays **OFF**.' docs/pages/scratch-tables.md | c7e6bcb | 'The host neither constructs a temporary sink nor uses the sink verification/transposition/staging files; it uses ordinary report arrays and the in-mem' docs/pages/scratch-tables.md | c7e6bcb | 'Optional recorded outputs use the ordinary retained-output reader.' docs/pages/scratch-tables.md | c7e6bcb | 'Keep the report sink OFF and use the ordinary report route.' docs/ci.md | 03b2406 | 'Push this prepared migration only after both CORE-TOP gates: laneboss posts **"pf-ci-release live"** and the owner answers **REL-HOSTED**.' docs/ci.md | 03b2406 | 'Until then a job on that label with no runners only waits, and nothing is pushed.' docs/ci.md | 03b2406 | 'Each job using these two CI fragments uses the expression verbatim and sets its matching array as a job-level **string** `PF_CI_LABELS`, merged into a' docs/ci.md | 03b2406 | 'The pre-job hook for those CI fragments also requires the matching `PF_CI_LABELS` string.' docs/ci.md | 03b2406 | 'It pins the hosted PR deployment job in `docs.yml`, the isolated jobs in `release.yml` and `promote-baseline.yml`, and the complete release matrix wit' docs/ci.md | 03b2406 | 'Both workflows require exact runner values without a fallback and carry no `PF_CI_LABELS`; unlisted credential-bearing jobs retain the existing hosted' docs/ci.md | 03b2406 | 'A credential-bearing job on a PR pool, a restored hosted label on a moved release job, a PR trigger on `release.yml`, a `pull_request` trigger or chan' Second list, 14 more deletions: the change is judged against main 9eff925 as one commit, so old main sentences that the 1.7.0 pages replace are deletions too. Introducers 4c78f39, d3cca9e, d093560, 47184e5, 41ca2ee, f48b7c8, a8f84b9, 65c69b0; same format. (Read by ORG-CX from the preflight of the previous single commit.) README.md | 4c78f39 | 'The counts below include recorded outputs and describe the released 1.5.0 API.' docs/design/native-feature-parity.md | d3cca9e | '`sha256:H` c_abi.cpp:N, `sha256:H` check_c_abi_runtime.py:N, c_abi.cpp:N, `sha256:H` c_abi.cpp:N, `sha256:H` c_abi.cpp:N, `sha256:H` c_abi.cpp:N, `sha' docs/native-refactor-progress.md | d093560 | '`WorkingRequestCore::history` (`include/pineforge/native_order.hpp:N`) hands out the whole vector and the core holds `history_` (`include/pineforge/na' docs/native-refactor-progress.md | d093560 | "`decompose_ms_local` (`src/session_time.cpp:N`) reads two thread-locals per call on a zoned clock, and the prepared order storage's `free_blocks` (`sr" docs/pages/index.md | 4c78f39 | 'The counts below include recorded outputs and describe the released 1.5.0 API.' docs/pages/index.md | 47184e5 | '- `<pineforge/execution_observer.h>` and `<pineforge/selected_window.h>` — three each: the six of the 80 runtime implementations that `pineforge.h` do' docs/pages/native-engine.md | 41ca2ee | '`closed_trade(i)` is the row itself, by reference — the object `get_trade(i)` and `get_report_trade(i)` read at the same index and the row `fill_repor' docs/pages/native-engine.md | f48b7c8 | "A run that began owns its rows whatever became of it: a `Completed` report and a started-then-failed run's partial rows are published unchanged." docs/pages/native-engine.md | d093560 | 'A C++ host reads a closed row whole, at any time, every `Trade` field included.' docs/pages/report-schema.md | a8f84b9 | '`pf_report_t` is the single output of every backtest.' docs/pages/report-schema.md | a8f84b9 | 'This page maps each field to its meaning and units.' docs/pages/report-schema.md | a8f84b9 | 'Heap array, ordered by exit time ascending.' docs/pages/run-failure-codes.md | 47184e5 | 'The other 38 codes retain their released 1.4.0 status.' docs/pages/run-failure-codes.md | 65c69b0 | 'Machine memory; retryable, capped.' Also in this commit: S4E's tag-time follow-up 6dfe9da (tests and two doc paragraphs; no src/ or include/ change). Its own message follows verbatim so scripts/check_doc_reverts.py reads the sentences it rewrites or qualifies in docs/design/o1-report-contract.md and docs/pages/run-failure-codes.md. tests, docs: REV's follow-ups on the scratch series: the sink_fd rows name their real subject and cause, the capability row's cells are meaningful and disclosed, the predicate scan covers the signed spellings, the docs state the ENOTSUP exception tests/test_run_json_sink_fd.py: the comment of the unjailed TMPDIR row now says the pre-series failure came from the native writer opening work/bad.sink in the mode-0555 work directory (sink_io), not from TMPDIR; new row test_native_fd_fsize_zero_path_sink_refused_in_read_only_work restores that subject (work 0555, TMPDIR 0755, report_sink_failed sink_io, nothing created); the TMPDIR-cause row runs without an arena, without PINEFORGE_COMMAND_SCRATCH_DIR and PINEFORGE_RUN_JSON_PROFILE, and proves DAC enforcement with an os.open probe first (visible pytest.skip with its reason otherwise). tests/test_default_scratch_effective_credentials.cpp nonroot_row() only (row() and main() unchanged): the setreuid outcome check can fail (EPERM or EINVAL accepted, credentials verified on both branches), the disclosure text is a pure fixture function pinned in test_scratch_capability_parse (every omitted cell "<cell> omitted (<reason>)", "missing, unreadable or malformed"), four more parser cases (0x80, -80, +80, space after the colon). tests/test_scratch_filesystem_predicate.py: the scan and its self-test also cover -2054924042(L) and 0xffffffff858458f6. docs/design/o1-report-contract.md and docs/pages/run-failure-codes.md: one paragraph each states the ENOTSUP exception (no probe, or a real errno 95 from stat/faccessat/statfs, keeps the default candidate as in 1.6.3 with no tmpfs claim and no note; /var/tmp selected without the rejected-candidate note when its probe answers ENOTSUP) and qualifies, with "except as described below", the sentences "A failed filesystem query stops default resolution with its syscall errno and names the failed candidate at every create wrapper; it never permits fallback." and "If creation in the selected `/var/tmp` fails, the detail names that directory, the rejected first candidate and `set TMPDIR to a disk-backed directory`." (o1-report-contract.md) and "A failed query stops resolution with its syscall errno and names the failed candidate." and "failures and all-memory refusals" (run-failure-codes.md); the other sentences keep their words and only their line wrapping changed (same number of lines in each block). No production code changed. Mutations m15 (setreuid EFAULT) and m16 (signed magic). Also in this commit, frozen 2026-10-24 for the 1.7.0 tag (three additions to the tree the CR-150 legs of 6f8f40e measured; src/ and include/ are unchanged): 1. CHANGELOG.md, section "Costs that remain": three new bullets (ORG-CX), 49 lines added and none removed, so no sentence is deleted. They state the costs that remain with the symbols they name: the I/O paths that can terminate or fail silently, the reads of closed trades and cohorts that now cost I/O, and the report-sink cost. 2. tests/test_scratch_arena.cpp (LAB-CX, commit 280c472 on parent 0f5072d, whose tree equals the one measured; test-only, +25/-4): on the CLOUD Batch VM the container filesystem answers open(O_TMPFILE) with errno 95 (EOPNOTSUPP), measured on lab job rj-20261024t121649-2fe44d, so the row failed there with the bare message "create backing descriptor" while it passes on NTU2 (rj-20261024t121045-c354ce). anonymous_regular_backing now falls back to mkstemp + unlink ONLY when the O_TMPFILE open fails with EOPNOTSUPP, so the anonymous regular-file arena shape is still exercised; it prints a NOTE with the errno when it does. Any other errno still fails, now with "errno N (text)" in the message. Nothing is skipped and no assertion is weakened: after the descriptor exists the row runs the same preallocation, attachment, ensure_size and zero-read checks as before. The engine never opens an O_TMPFILE. Docs addition 3 (ORG-CX R9-F1/F2, docs only, line counts unchanged): three public pages stop promising an envelope run_json never emits and a NULL sink removal that can be refused. Its own message follows verbatim so scripts/check_doc_reverts.py reads the nine sentences it replaces. docs: three public pages stop promising an envelope run_json never emits and a NULL sink removal that can be refused REV's read of the frozen tree (R9-F1, R9-F2): the code and the CHANGELOG are right, three pages are not. report-schema.md (lines 253-267) said explicit-mode JSON carries `report_contract`/`report_capture` and that explicit modes add policy keys and move the fingerprint to version 3. On the `run_json.py` transport an explicit `--report-mode FULL` is the legacy route (docker/run_json.py: "--report-mode FULL is identical to omitting the option"; only SUMMARY_ONLY and STREAMED are refused as `report_mode_unsupported`), so it adds none of those keys; they appear on a host that fills the v1 envelope (C++ `fill_report_envelope_v1`). o1-report-contract.md (lines 47-55) made the same claim for every explicit request. abi-stability.md (lines 165-172) listed -1, -2 and -3 for the sink setters but not that NULL returns -3 while an idle handle still holds a completed run's scratch curve or sealed rows (src/engine_report.cpp set_report_sink_v1 and set_report_sink_fd_v1), that an invalid scratch-arena environment returns -1, or that a completed fd generation detaches and keeps its receipt and summary (include/pineforge/report_sink.h documents all three). Line counts of all three pages are unchanged (818, 584, 672), so no citation moves. Sentences replaced (named for the doc-reverts check): - "Explicit-mode JSON includes `report_contract: "pineforge-report/v1"` and `report_capture`, containing `version`, the mode name, `complete`, and all seven sequence objects above." - "Explicit modes also add `report_contract`, `report_mode`, and `sequence_capture` to `applied_runtime` and fingerprinted `provenance.runtime`." - "Either sink setter replaces the one installed sink, and NULL through either setter removes it." - "Both return 0 on acceptance, -1 for a bad handle or descriptor prefix (or a negative fd), -2 during a stream and -3 during a batch run or report capture." - "Run-begin validation and write errors use the existing typed `report_sink_failed` path." - "The caller keeps the fd open through execution and report capture, then closes it to signal EOF." - "The engine never duplicates or closes a borrowed fd, including on failure, replacement or handle destruction." - "No option keeps the legacy route." - "An explicit request must either return the requested versioned envelope or fail." Frozen2 additions (2026-10-24, after REV's P1 on the adapter retirement sweep, R6a-F1): the tree of the first frozen commit cad8b4b plus exactly four changes, nothing else. 1. src/source/pine_adapter.cpp (CR150-CX, commit f5deaf8, parent cad8b4b, 6 insertions and 6 deletions): erase_retired_rows keeps a dormant askable-origin exit leg pinned for its cycle, as v1.6.3's K1 pinned it. The retirement sweep released a dormant askable-origin exit leg A when a later same-family, same-source, same-from_entry exit with different stop bits existed (release_superseded), dropped it from the K2/K3 candidates, and let K3 keep it anyway as the first witness of a live exit Y. A's own supersessor B (released by a later C, nobody's witness, no projection_predecessor link) was then erased. revive_brackets_after_margin decides "superseded" only over retained rows, so with B gone and C's stop bits equal to A's, a margin call restored A as a marketable flatten at the margin price under A's exit id: an extra trade that v1.6.3 never books (v1.6.3 pinned every unwithdrawn lifecycle row for the cycle). held_by_cycle and the matching collection of held legs in the unpinned pre-pass now require the leg's origin NOT to be askable, dormant or not: a dormant askable-origin leg is pinned by pinned_by_itself exactly as in v1.6.3 and is never released. Withdrawn legs, non-askable origins and everything else are unchanged. The cost: a script that keeps re-issuing strategy.exit(qty=) over a dormant askable leg without cancelling retains its rows while the origin stays open, as v1.6.3 did. 2. tests/test_k1_askable_origin_supersessor_erasure.cpp and tests/cr150.d/p2cx-k1.cmake (P2-CX, commit b4c5a42, parent cad8b4b; the registration is the tree's own tests/cr150.d/*.cmake glob, so a clean checkout builds it): the K1 reproduction of REV's R6a-F1 tape runs on two fresh hosts, default ERASE and RETAIN-every-row. Per-bar leg, dormancy and position state is printed for both, controls assert that each tape condition held, and the only oracle is erase trades == retain trades (no literal trade list). A red here (erase books an extra margin-price flatten under "X") proves REV's P1. Green on NTU2 with the fix: job rj-20261024t140803-0bf6e2, 8/8, ERASE == RETAIN. 3. CHANGELOG.md (ORG-CX, patch CHANGELOG-171-DORMANT, sha256 1407fc7f980338d4b11ef31ab25aac5b7df3a9a299e8db09757307b708b0b610): a new last bullet of "Costs that remain", "Re-issued exits on a dormant leg keep one placement row per re-issue" (card ORG-171-DORMANT), and one exception clause in the "placement retirement rules (CR-150)" entry. The sentence it changes (named for scripts/check_doc_reverts.py): "for the dormant exit legs of one origin, exit id, family and entry only the oldest is kept as the representative, with superseded dormant legs released; the retired rows leave the hashed placement sequence, so hash values change wherever such a placement retires;" now reads "..., with superseded dormant legs released (except a dormant leg that was not withdrawn while its origin is still open: it is kept until its position cycle ends, see Costs that remain); the retired rows leave the hashed placement sequence, so hash values change wherever such a placement retires;". 4. The message of the first frozen commit is carried verbatim above, with ONE number corrected: "Line counts of all three pages are unchanged (818, 584, 672), so no citation moves." (the first message said 817; docs/pages/report-schema.md has 818 lines). Frozen3-MIN addition (2026-10-25, CORE-TOP/L6 rulings of 01:51 and 02:14): the tree of the second frozen commit c82fd00 plus exactly the inputs below, nothing else. No source or header file other than the three named in paragraph 5 changes relative to c82fd00. 5. include/pineforge/engine.hpp, include/pineforge/trade_ledger.hpp, src/engine_orders.cpp, tests/cr150.d/p2cx-tledger.cmake, tests/p2cx_tledger_fixture.hpp, tests/test_p2cx_tledger_budget.cpp, tests/test_p2cx_tledger_io.cpp (P2-CX, the cumulative patch of commits 6548775 and d21ea8d on c82fd00, sha256 05b77f191b974e09f028a6b0d9942798ec1a44e20c9de4410866e7e01cd5d3e6, 1181 insertions and 46 deletions; the tests register through the tree's own tests/cr150.d/*.cmake glob): TradeLedger gets a budgeted read cache of decoded closed rows and per-field accessors. A script that reads strategy.closedtrades every bar decoded each historical row per field call (an index pread below the 64-row descriptor window and four string preads outside the 10752-byte payload tail) and every Pine accessor built a whole Trade by value to read one field. Now TradeLedger::<field>_at(i) returns one field straight from the cached record (numeric fields build no Trade and no string; string fields read only the requested packed bytes), with the same checked-row contract as at(); the closed_trade_* accessors in engine.hpp use them, with values, error behaviour and percent formulas unchanged. The cache is direct-mapped by global index; the default budget is 65,536 rows (maximum 1,048,576); one entry is a 168-byte record plus 16 bytes of key and generation plus 128 inline string bytes, 312 bytes (20,447,232 bytes at the default); a row whose four strings exceed 128 bytes keeps its numeric fields cached and reads strings from scratch; the array grows lazily to min(budget, next power of two >= max(64, rows)), so memory is bounded by the budget independent of bars. seal() no longer invalidates the cache (publication changes no row content and the checked accessors refuse sealed rows before reaching the cache); replace, rewind (resize), move and clear invalidate it; replace() reuses an unchanged string extent instead of appending the same bytes again. kResidentRows (64), kRecordBytes, resident_size() and resident_base() are unchanged. No environment variable or run-spec field controls the budget: the default is what runs (C++ API only). The budget setter's refusal still uses setting_rejected until the shared resident-budget code lands. Tests: test_trade_ledger, test_p2cx_tledger_io (syscall counter, zero reads and writes in a hungpixi-shaped 24,238-row loop after warm-up, oracle-compared) and test_p2cx_tledger_budget (budget formula, lazy growth boundaries, wide-string fallback, replace, rewind and move invalidation, sealed-row refusal for every accessor); P2-CX reports them Passed on NTU2 (job rj-20261024t175023-a20f66) with mutation rows red on the intended assertions. SM-CX measured this patch on frozen2: hungpixi 106.14 to 107.58 s over four repetitions against 72.84 to 73.55 s on 1.6.3, trades identical in every cell; the engine still makes scratch reads and writes per bar on other paths (id tables, exit scratch, journal), which is the cost named in the CHANGELOG. 6. CHANGELOG.md and documentation pages (ORG-CX; patches CHANGELOG-171-DORMANT-V2, OVERCLAIM-170-v4, REPIN-FROZEN3MIN-NOPAGE, CHANGELOG-170-SLOWER-FILLED-v3 and CHANGELOG-LEDGERCACHE-v2, applied in that order after paragraph 5; CHANGELOG-LEDGERCACHE-v2 rewrites the opening of the "Reads of closed trades and cohorts now cost I/O" bullet because the closed-trade read cache of paragraph 5 makes its statement about scratch reads on every call false: it now states the cache, its default budget of 65,536 rows of 312 bytes (20,447,232 bytes at most per engine), its lazy allocation at the first closed trade of every Pine run, that above 65,536 closed trades it is direct-mapped so a loop over the whole history re-reads colliding rows from scratch files on every pass, and that no user control sets it in this release): the dormant-leg bullet of "Costs that remain" is rewritten (V2), nine over-broad memory sentences are corrected (v4), the engine.hpp citation numbers in docs/adr/0001-kernel-adapter-boundary.md, docs/design/native-feature-parity.md, docs/pages/native-engine.md and docs/pages/pine-to-native.md are re-pinned by the anchor tool because paragraph 5 moves engine.hpp lines by one or two (the box anchor check reports every documentation anchor resolved), and a new first bullet of "Costs that remain" states that 1.7.0 is slower than 1.6.3 on some scripts with the figures SM-CX measured (no existing sentence is replaced by that bullet). No memory-management page is added. The sentences replaced or deleted by this paragraph, named for scripts/check_doc_reverts.py: - "The bounded paths of this release are the trade ledger, the report sink and the command journals described above." (CHANGELOG.md, Costs that remain) - "The complexity in the number of rows is unchanged; the cost per row is not." (CHANGELOG.md, Costs that remain) - "Origin/cohort receipts, bracket rosters, historical IDs, successor roots and position tails use retirement, bounded resident windows or scratch instead of retaining all history in RAM." (CHANGELOG.md, Adapter history retirement) - "the production Pine host uses KernelRecordedAtHostMarks (src/source/pine_adapter.cpp:2870), whose recorded curve is held in O(1) and streamed to the scratch file." (docs/design/cr150-contracts.md) - "Complete enabled sequences, in existing report order, through a bounded sink" (docs/pages/report-schema.md, STREAMED row) - "Fixed slab for historical table bytes, prepared before the run, with count/byte admission and no growth or disk fallback on exhaustion." (docs/pages/scratch-tables.md, bounded_memory row) - "For the short waiting-child parent-terminal fixture, the book's 4,096-row cap and the journal's 2,048-event cap give a per-engine definition plateau of 1,890,980 bytes (about 1.9 MB) on GNU 13.4 x86-64, independent of symbols, window and trades." (docs/pages/scratch-tables.md) - "caps (book 4,096 + journal 2,048 events), constant, independent of symbols, window and trades." (docs/scratch-store.md, resident bound of the parent-terminal drain) - "for the dormant exit legs of one origin, exit id, family and entry only the oldest is kept as the representative, with superseded dormant legs released; the retired rows leave the hashed placement sequence, so hash values change wherever such a placement retires;" (CHANGELOG.md) - "**Reads of closed trades and cohorts now cost I/O.** The prefix sums described above apply to sealed rows only. Unless history is streamed, the trade ledger keeps 64 rows resident and reads older closed rows, text included, back from scratch files on every call, where the previous release walked an in-memory vector (`std::vector<Trade>`)." (CHANGELOG.md, Costs that remain, closed-trade reads bullet; rewritten by CHANGELOG-LEDGERCACHE-v2) - "`BacktestEngine::avg_trade_percent`, `avg_winning_trade_percent`, `avg_losing_trade_percent` and the `closed_trade_*` accessors therefore read every older row per call, so a script that calls them each bar pays one scratch read per row per call." (CHANGELOG.md, same bullet; rewritten by CHANGELOG-LEDGERCACHE-v2) Every citation line changed by the anchor re-pin differs from main only in a line number inside the same sentence; the doc-reverts checker normalises numbers, so none of them is named above. The message of the second frozen commit is carried verbatim above (frozen2 paragraphs 1 to 4, which in turn carry the first frozen commit's message). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 9eff925 commit 69467f7

690 files changed

Lines changed: 207630 additions & 5323 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,7 @@ scripts/
2727
tutorial/
2828
.playwright-mcp/
2929
node_modules/
30+
31+
# Frozen Python report producer used only by the L9A comparison tests.
32+
tests/fixtures/run_json_frozen_pre_l9a.py
33+
tests/fixtures/run_json_frozen_pre_l9a.sha256

‎.github/workflows/ci-post.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,7 @@ jobs:
311311
if: ${{ needs.authz.result == 'success' && runner.os == 'Linux' }}
312312
run: |
313313
sudo apt-get update
314-
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libeigen3-dev tzdata-legacy ccache
314+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libeigen3-dev tzdata-legacy ccache strace
315315
316316
# PRs restore the newest compatible main cache via the default-branch
317317
# fallback. Only main writes this namespace, never a PR merge ref.

‎.github/workflows/native-live.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ jobs:
145145
- name: Install native build dependencies
146146
run: |
147147
sudo apt-get update
148-
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y cmake ninja-build libeigen3-dev tzdata-legacy libsqlite3-dev libssl-dev zlib1g-dev ccache
148+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y cmake ninja-build libeigen3-dev tzdata-legacy libsqlite3-dev libssl-dev zlib1g-dev ccache strace
149149
150150
- name: Restore ccache
151151
id: ccache-restore

‎CHANGELOG.md‎

Lines changed: 754 additions & 12 deletions
Large diffs are not rendered by default.

‎CMakeLists.txt‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,12 +97,15 @@ set(PINEFORGE_SOURCE_LAYER_SOURCES
9797
src/compat/pine/reservation_expansion.cpp
9898
src/source/magnifier_intrabars.cpp
9999
src/source/market_admission.cpp
100+
src/source/admission_journal_codec.cpp
100101
src/source/pine_adapter.cpp
102+
src/source/pine_position_tail.cpp
101103
src/source/pine_path_resolve.cpp
102104
src/source/pine_scheduler_native.cpp
103105
src/source/pine_scheduler.cpp
104106
src/source/pine_strategy_host.cpp
105107
src/source/pine_strategy_commands.cpp
108+
src/source/exit_scratch_versions_production.cpp
106109
src/source/pine_value_admission.cpp
107110
src/source/pine_aux_security.cpp
108111
src/source/pine_security_eval.cpp
@@ -131,9 +134,12 @@ set(PINEFORGE_KERNEL_SOURCES
131134
src/engine_trade_accessors.cpp
132135
src/pending_order_mirror.cpp
133136
src/native_order.cpp
137+
src/native_journal_codec.cpp
134138
src/engine_consumer.cpp
135139
src/market_driver.cpp
136140
src/native_execution_consumer.cpp
141+
src/command_scratch_production.cpp
142+
src/command_scratch_cache.cpp
137143
src/native_calc_plan.cpp
138144
src/magnifier.cpp
139145
src/math.cpp
@@ -148,13 +154,18 @@ set(PINEFORGE_KERNEL_SOURCES
148154
src/timeframe.cpp
149155
src/timezone.cpp
150156
src/native_calendar.cpp
157+
src/report_sink_writer.cpp
158+
src/sha256.cpp
151159
src/native_run_spec.cpp
152160
src/native_fx_curve.cpp
161+
src/native_bar_views.cpp
153162
src/selected_report_view.cpp
154163
src/selected_window_plan.cpp
164+
src/metrics_spill_curve.cpp
165+
src/report_scratch.cpp
166+
src/scratch_arena.cpp
155167
)
156-
# The kernel-only archive. Same objects, same flags as the kernel half of
157-
# libpineforge.a — a consumer that never speaks Pine links this one.
168+
# The kernel-only archive: libpineforge.a's kernel objects and flags, for consumers that never speak Pine.
158169
add_library(pineforge_kernel STATIC ${PINEFORGE_KERNEL_SOURCES})
159170
add_library(PineForge::kernel ALIAS pineforge_kernel)
160171
# So a find_package(PineForge) consumer spells it the same way an in-tree one

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ which needs SQLite3, libcurl ≥ 7.86 and OpenSSL. On Debian/Ubuntu also
9292
install `tzdata-legacy` (`sudo apt-get install -y tzdata-legacy`): noble split
9393
the backward zone links out of `tzdata`, and `test_native_calendar` resolves
9494
`US/Eastern`, `Japan` and `GB`.
95+
Linux hosts running `test_run_json_jailed_nofile_e2e` require `strace` with permission to trace child processes (ptrace).
9596

9697
> **The stale-test-binary trap.** `cmake --build build --target pineforge`
9798
> rebuilds only the static library. Test executables are separate targets that

‎README.md‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3434
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies), measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
36-
- **Yours to embed.** 145 `PF_API` declarations across five installed headers — `pineforge.h`'s 94 (74 of the 80 runtime exports, the other six being declared in `execution_observer.h` and `selected_window.h`, three each, and up to 20 per-strategy functions a generated module defines), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
36+
- **Yours to embed.** 158 `PF_API` declarations across six installed headers — `pineforge.h`'s 102 (82 of the 93 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3737

3838
Recorded outputs and their additional API declarations described here are
3939
part of **1.5.0**.
@@ -58,7 +58,7 @@ cases, and no completeness claim (the published 1.6.0 image declares `1.3`).
5858
`capabilities.selected_window_contract_revisions` is `["1.3-parity"]` when the
5959
runtime supports selected windows and `[]` otherwise.
6060

61-
From 1.6.2 (unreleased), the engine/codegen image advertises selected-window
61+
From 1.6.2, the engine/codegen image advertises selected-window
6262
contract revision `1.3` through `docker/run_json.py --capabilities-json`:
6363
with the section-4 producers and manifest present and value ownership
6464
`complete` (EN-115), it emits the four typed codes (`strategy_value_invalid`,
@@ -536,13 +536,13 @@ API, the script ABI epoch and the pairing with codegen — is the
536536
[public contract](docs/pages/public-contract.md); what a 0.x user must act on
537537
is in [CHANGELOG.md](CHANGELOG.md).
538538

539-
The counts below include recorded outputs and describe the released 1.5.0 API.
539+
The counts below describe the current API, including recorded outputs, the report sink and scratch-table selection; recorded outputs are not part of 1.4.0.
540540

541-
A newly generated strategy `.so` exposes 96 compiled-strategy `PF_API` declarations
542-
(80 runtime implementations plus sixteen generated exports) plus 43 native-host
543-
declarations: 139 `PF_API` exports in total; `use_bar_magnifier = true` adds
541+
A newly generated strategy `.so` exposes 109 compiled-strategy `PF_API` declarations
542+
(93 runtime implementations plus sixteen generated exports) plus 43 native-host
543+
declarations: 152 `PF_API` exports in total; `use_bar_magnifier = true` adds
544544
`strategy_declares_bar_magnifier` and a module that records [outputs](docs/outputs.md) its
545-
three outputs exports, bringing the inventories to at most 100 and 143. Older modules lack the six opt-in
545+
three outputs exports, bringing the inventories to at most 113 and 156. Older modules lack the six opt-in
546546
[checked-settings exports](docs/checked-settings.md). Older modules can also lack
547547
the two opt-in [execution-capability exports](docs/strategy-capabilities.md).
548548
In an optimized build `nm -gU` also shows libc++'s
@@ -567,13 +567,18 @@ inventories are pinned by `scripts/check_c_abi_runtime.py`:
567567
| `strategy_set_chart_timezone` / `strategy_set_syminfo_timezone` / `strategy_set_syminfo_session` | Chart and exchange time |
568568
| `strategy_set_syminfo_mintick` / `_pointvalue` / `_metadata` / `_type` / `_string` | Symbol tick size, point value, numeric metadata, instrument class, string members |
569569
| `strategy_set_native_security_feed` / `strategy_set_aux_security_feed` | Feed `request.security()` from a native higher-timeframe series / an auxiliary bar-aligned feed |
570-
| `strategy_set_symbol_feed` / `_feed_column` / `strategy_set_symbol_facts` / `strategy_set_recorded_series` | Another symbol's data for `request.security()` of that symbol: its own bars (each with its close) and named columns, its `syminfo.*` facts, and recorded request values per chart bar (each behind its own `PINEFORGE_HAS_…_V1` probe; historical runs only) |
570+
| `strategy_set_symbol_feed` / `strategy_set_symbol_feed_borrowed_v1` / `_feed_column` / `strategy_set_symbol_facts` / `strategy_set_recorded_series` | Another symbol's data for `request.security()` of that symbol: its own copied or borrowed bars (each with its close) and named columns, its `syminfo.*` facts, and recorded request values per chart bar (each behind its own `PINEFORGE_HAS_…_V1` probe; historical runs only) |
571571
| `strategy_set_account_currency_fx_series` | Effective-time quote-to-account FX |
572572
| `strategy_get_last_error` | The latest runtime error |
573573
| `strategy_get_last_error_code` / `strategy_get_last_error_args` | The latest failure's stable code from the closed [run-failure vocabulary](docs/pages/run-failure-codes.md) (`""` when the run succeeded) and its typed arguments as canonical JSON; behind `PINEFORGE_HAS_RUN_FAILURE_CODES_V1` |
574+
| `pf_report_scratch_arena_stats` | Process-wide scratch-arena peak claimed bytes and exact supplied size; query after finalization and before detach for stdout diagnostics, outside sink bytes |
575+
| `strategy_require_report_scratch_arena_v1` | Jailed startup admission: attach/validate the inherited fd-5 arena, or refuse with typed `report_sink_failed/arena_backing`; no temporary-file fallback |
576+
| `strategy_report_sink_diagnostics_v1` | Fixed-size presented equity endpoints and first-bar trade provenance after a completed sink; stdout annualization needs no sink read-back |
574577
| `pf_version_get` / `pf_version_string` / `pf_abi_version` | Runtime version, version string, struct-layout version (`PF_ABI_VERSION == 4`) |
575578
| `strategy_execution_contract` / `strategy_configure_native_v1` / `strategy_configure_native_fx_curve_v1` / `_fx_curve_ext_v1` | Query Legacy vs NativeMarketV1; apply the versioned native run specification (an invalid or refused base call returns `-1` and leaves the handle Failed; a cooperative `Aborted` handle may be reused with the same key and a higher run number); stage or clear an immutable native FX curve, `_fx_curve_ext_v1` also writing the typed refusal (`pf_native_fx_curve_error_e`) and the offending point's index |
576579
| `strategy_request_abort` / `strategy_last_run_status` | Cooperative abort of a run in progress; `0`=completed, `1`=aborted |
580+
| `strategy_set_scratch_tables_v1` | Copy disk-scratch or bounded-memory table policy and its eleven admission maxima while idle; NULL restores disk defaults (since 1.7.0) |
581+
| `pf_run_abort_install_sigterm_v1` / `pf_run_abort_requested_v1` / `pf_run_abort_restore_sigterm_v1` | Opt-in child invocation SIGTERM flag; stop after a completed input bar, retaining typed progress and an incomplete sink; install/restore with SIGTERM blocked |
577582
| `strategy_set_realtime_tail` | Live-runtime surface (ABI v4): the array's last bar is a still-forming tail — `barstate.islast=false`, `last_bar_index`/`last_bar_time` frozen at the horizon bar, no range-end row |
578583
| `strategy_set_probe_suppress_tail_logic` | ABI v4: the last bar runs only the broker's pre-`on_bar` steps (pending-order settlement, intraday-cap/loss checks) and returns — no `on_bar`, no margin-call / `process_orders_on_close` second pass / bracket-reissue processing (the range-end row is `strategy_set_realtime_tail`'s to skip; the flags are independent) |
579584
| `strategy_set_path_order` / `strategy_last_bar_dual_entry_path` | ABI v4: force the intrabar O→H/L→C leg order (`AUTO`/`HIGH_FIRST`/`LOW_FIRST`) for path-dependent fill probing; read which side won a same-bar dual-entry-stop arbitration |

‎benchmarks/throughput/grid_search_repro.py‎

Lines changed: 46 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,14 @@
22
import ctypes
33
import csv
44
import itertools
5+
import sys
56
from pathlib import Path
67

8+
# Share the host's descriptor and policy arithmetic, including typed refusals.
9+
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "docker"))
10+
from run_json import (RunFailure, _bind_scratch_tables, _check_bench_run,
11+
apply_scratch_tables, default_scratch_manifest, failure_line)
12+
713
class pf_bar_t(ctypes.Structure):
814
_fields_ = [
915
("open", ctypes.c_double),
@@ -192,6 +198,16 @@ def main():
192198
lib.report_free.argtypes = [ctypes.POINTER(pf_report_t)]
193199
lib.report_free.restype = None
194200

201+
_bind_scratch_tables(lib)
202+
for name in ("strategy_get_last_error", "strategy_get_last_error_code",
203+
"strategy_get_last_error_args"):
204+
if hasattr(lib, name):
205+
getattr(lib, name).argtypes = [ctypes.c_void_p]
206+
getattr(lib, name).restype = ctypes.c_char_p
207+
if hasattr(lib, "strategy_last_run_status"):
208+
lib.strategy_last_run_status.argtypes = [ctypes.c_void_p]
209+
lib.strategy_last_run_status.restype = ctypes.c_int
210+
195211
# Load CSV
196212
bars_list = []
197213
print(f"Loading CSV data from {csv_path.name}...")
@@ -230,29 +246,40 @@ def main():
230246
print("Failed to create strategy instance!")
231247
continue
232248

233-
# Set inputs
234-
lib.strategy_set_input(s, b"Fast EMA", str(fast).encode())
235-
lib.strategy_set_input(s, b"Slow EMA", str(slow).encode())
236-
lib.strategy_set_input(s, b"Take profit (ticks)", str(tp).encode())
237-
238249
report = pf_report_t()
239-
lib.run_backtest(s, bars_array, n, ctypes.byref(report))
240-
241-
profit = report.net_profit
242-
trades = report.total_trades
243-
244-
print(f"Params: Fast_EMA={fast}, Slow_EMA={slow}, TP_ticks={tp} | Net Profit = {profit:.2f} | Trades = {trades}")
245-
246-
if profit > best_profit:
247-
best_profit = profit
248-
best_params = (fast, slow, tp)
249-
250-
lib.report_free(ctypes.byref(report))
251-
lib.strategy_free(s)
250+
try:
251+
lib.strategy_set_input(s, b"Fast EMA", str(fast).encode())
252+
lib.strategy_set_input(s, b"Slow EMA", str(slow).encode())
253+
lib.strategy_set_input(s, b"Take profit (ticks)", str(tp).encode())
254+
255+
# P0 policy for THIS trial, after its overrides/feed selection.
256+
# A fresh handle/run receives fresh occupancy, never its predecessor's.
257+
manifest = default_scratch_manifest(n)
258+
apply_scratch_tables(lib, s, manifest)
259+
lib.run_backtest(s, bars_array, n, ctypes.byref(report))
260+
_check_bench_run(lib, s) # a failed trial cannot enter the ranking
261+
262+
profit = report.net_profit
263+
trades = report.total_trades
264+
265+
print(f"Params: Fast_EMA={fast}, Slow_EMA={slow}, TP_ticks={tp} | Net Profit = {profit:.2f} | Trades = {trades}")
266+
267+
if profit > best_profit:
268+
best_profit = profit
269+
best_params = (fast, slow, tp)
270+
finally:
271+
try:
272+
lib.report_free(ctypes.byref(report))
273+
finally:
274+
lib.strategy_free(s)
252275

253276
print("\n=== OPTIMIZATION COMPLETE ===")
254277
print(f"Best Profit: {best_profit:.2f}")
255278
print(f"Best Parameters: Fast EMA = {best_params[0]}, Slow EMA = {best_params[1]}, Take profit (ticks) = {best_params[2]}")
256279

257280
if __name__ == "__main__":
258-
main()
281+
try:
282+
main()
283+
except RunFailure as failure:
284+
sys.stderr.write(failure_line(str(failure), failure.code, failure.code_args))
285+
sys.exit(failure.exit_status)

‎corpus‎

Submodule corpus updated 313 files

0 commit comments

Comments
 (0)