Repository navigation
CR-150 O(1) report integration: the engine 1.7.0 line (final6: + sink-run ledger cache bound) - #499
Merged
Merged
Conversation
…ver main 9eff925 One squash-equivalent commit: parent = engine main 9eff925 (#463), tree = the CR-150 picks line merged with that main (docs/ci.md the only conflict: the line's clarification paragraph and main's two baseline-promotion paragraphs are adjacent additions, both kept). The squash body at the merge carries this message verbatim. The 40 documentation sentences below were deleted by commits of the line and are intended: they are the superseded wording of the pages the final 1.7.0 text replaces. scripts/check_doc_reverts.py reads this message for them. Format per line: file | introducer short sha | start of the deleted sentence. Introducers: dbf56ed (docs/design/arena-bound.md 22), 03b2406 (docs/ci.md 7), c7e6bcb (docs/pages/scratch-tables.md 5), 1fa26e7 (scratch-tables.md 3), b4a3f76 (scratch-tables.md 1), 00dec17 (docs/scratch-store.md 1), 2688775 (docs/pages/abi-stability.md 1). docs/pages/abi-stability.md | 2688775 | "The engine validates/attaches the inherited fixed-fd-5 arena or returns the handle's typed `report_sink_failed`/`arena_backing`." docs/pages/scratch-tables.md | 1fa26e7 | 'A jailed arena must be a sealed memfd with at least `F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL`, and the jailed admission (`strategy_require_report_sc' docs/pages/scratch-tables.md | 1fa26e7 | 'Where sealed memfds do not exist (not Linux) the admission accepts the arena the host supplies.' docs/pages/scratch-tables.md | 1fa26e7 | 'With `PF_SCRATCH_ARENA_FD` set, only exactly `"1"` enables it; unset or exactly `"0"` preserves the unjailed backing rules, and every other value (inc' docs/pages/scratch-tables.md | b4a3f76 | 'A jailed arena must be a sealed memfd with at least `F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL`; ordinary files, sealing-disabled memfds and unsupport' docs/pages/scratch-tables.md | c7e6bcb | 'The engine never changes process signal disposition.' docs/scratch-store.md | 00dec17 | 'Below it the drain is held to 4,096 B for every population up to 8,191, where the resident journal events own the definitions the spilled book rows ne' docs/design/arena-bound.md | dbf56ed | 'L18 sibling footer evidence described below; absent from this base' docs/design/arena-bound.md | dbf56ed | '`erased_minima_` `include/pineforge/source/pine_adapter.hpp:N`;' docs/design/arena-bound.md | dbf56ed | '`remember_erased` `src/source/pine_adapter.cpp:N`' docs/design/arena-bound.md | dbf56ed | '`consumed_partial_exit_cycles_` `include/pineforge/source/pine_adapter.hpp:N`;' docs/design/arena-bound.md | dbf56ed | '`src/source/pine_adapter.cpp:N`' docs/design/arena-bound.md | dbf56ed | 'The exact external file:line evidence is recorded in the lane report; these sources must be merged and their local anchors refreshed before claiming c' docs/design/arena-bound.md | dbf56ed | 'Summing all eleven peaks gives:' docs/design/arena-bound.md | dbf56ed | 'There are eleven persistent regions and up to three extra replacement indexes.' docs/design/arena-bound.md | dbf56ed | 'Each independently rounded region wastes less than C.' docs/design/arena-bound.md | dbf56ed | 'Summing independent peaks and adding fourteen complete chunks is conservative even when their real peak times differ.' docs/design/arena-bound.md | dbf56ed | 'This covers chunk fragmentation, not unlisted stores or a missing population proof.' docs/design/arena-bound.md | dbf56ed | 'At L=64 the position term is **94*T**; each placement data term is **104*D+168*Q**; consumed-cycle data is **169*Q**; the root term is **16*R**.' docs/design/arena-bound.md | dbf56ed | "The contract's eleven coefficients agree with the inspected code and L19l-A/C reports." docs/design/arena-bound.md | dbf56ed | 'No numeric correction to those coefficients was needed. “72..104 bytes/key” in the L19l-C steady-state explanation does not include the rehash overlap' docs/design/arena-bound.md | dbf56ed | 'Each of two placement data files' docs/design/arena-bound.md | dbf56ed | 'Each of two placement index peaks' docs/design/arena-bound.md | dbf56ed | 'Eleven contributions in the table above, including both two-file rows.' docs/design/arena-bound.md | dbf56ed | 'D=32,Q=1,L=1,E=1; both 128-slot rehash peaks are 3072;' docs/design/arena-bound.md | dbf56ed | 'No production header declaration is changed.' docs/design/arena-bound.md | dbf56ed | 'Linux linker wrappers reject write/pwrite/ftruncate on unlinked regular scratch files, while allowing the named report sink.' docs/design/arena-bound.md | dbf56ed | 'The current base still has those owning-port dependencies.' docs/design/arena-bound.md | dbf56ed | 'It must not be reported as corpus proof until they land.' docs/pages/scratch-tables.md | c7e6bcb | 'The report sink stays **OFF**.' docs/pages/scratch-tables.md | c7e6bcb | 'The host neither constructs a temporary sink nor uses the sink verification/transposition/staging files; it uses ordinary report arrays and the in-mem' docs/pages/scratch-tables.md | c7e6bcb | 'Optional recorded outputs use the ordinary retained-output reader.' docs/pages/scratch-tables.md | c7e6bcb | 'Keep the report sink OFF and use the ordinary report route.' docs/ci.md | 03b2406 | 'Push this prepared migration only after both CORE-TOP gates: laneboss posts **"pf-ci-release live"** and the owner answers **REL-HOSTED**.' docs/ci.md | 03b2406 | 'Until then a job on that label with no runners only waits, and nothing is pushed.' docs/ci.md | 03b2406 | 'Each job using these two CI fragments uses the expression verbatim and sets its matching array as a job-level **string** `PF_CI_LABELS`, merged into a' docs/ci.md | 03b2406 | 'The pre-job hook for those CI fragments also requires the matching `PF_CI_LABELS` string.' docs/ci.md | 03b2406 | 'It pins the hosted PR deployment job in `docs.yml`, the isolated jobs in `release.yml` and `promote-baseline.yml`, and the complete release matrix wit' docs/ci.md | 03b2406 | 'Both workflows require exact runner values without a fallback and carry no `PF_CI_LABELS`; unlisted credential-bearing jobs retain the existing hosted' docs/ci.md | 03b2406 | 'A credential-bearing job on a PR pool, a restored hosted label on a moved release job, a PR trigger on `release.yml`, a `pull_request` trigger or chan' Second list, 14 more deletions: the change is judged against main 9eff925 as one commit, so old main sentences that the 1.7.0 pages replace are deletions too. Introducers 4c78f39, d3cca9e, d093560, 47184e5, 41ca2ee, f48b7c8, a8f84b9, 65c69b0; same format. (Read by ORG-CX from the preflight of the previous single commit.) README.md | 4c78f39 | 'The counts below include recorded outputs and describe the released 1.5.0 API.' docs/design/native-feature-parity.md | d3cca9e | '`sha256:H` c_abi.cpp:N, `sha256:H` check_c_abi_runtime.py:N, c_abi.cpp:N, `sha256:H` c_abi.cpp:N, `sha256:H` c_abi.cpp:N, `sha256:H` c_abi.cpp:N, `sha' docs/native-refactor-progress.md | d093560 | '`WorkingRequestCore::history` (`include/pineforge/native_order.hpp:N`) hands out the whole vector and the core holds `history_` (`include/pineforge/na' docs/native-refactor-progress.md | d093560 | "`decompose_ms_local` (`src/session_time.cpp:N`) reads two thread-locals per call on a zoned clock, and the prepared order storage's `free_blocks` (`sr" docs/pages/index.md | 4c78f39 | 'The counts below include recorded outputs and describe the released 1.5.0 API.' docs/pages/index.md | 47184e5 | '- `<pineforge/execution_observer.h>` and `<pineforge/selected_window.h>` — three each: the six of the 80 runtime implementations that `pineforge.h` do' docs/pages/native-engine.md | 41ca2ee | '`closed_trade(i)` is the row itself, by reference — the object `get_trade(i)` and `get_report_trade(i)` read at the same index and the row `fill_repor' docs/pages/native-engine.md | f48b7c8 | "A run that began owns its rows whatever became of it: a `Completed` report and a started-then-failed run's partial rows are published unchanged." docs/pages/native-engine.md | d093560 | 'A C++ host reads a closed row whole, at any time, every `Trade` field included.' docs/pages/report-schema.md | a8f84b9 | '`pf_report_t` is the single output of every backtest.' docs/pages/report-schema.md | a8f84b9 | 'This page maps each field to its meaning and units.' docs/pages/report-schema.md | a8f84b9 | 'Heap array, ordered by exit time ascending.' docs/pages/run-failure-codes.md | 47184e5 | 'The other 38 codes retain their released 1.4.0 status.' docs/pages/run-failure-codes.md | 65c69b0 | 'Machine memory; retryable, capped.' Also in this commit: S4E's tag-time follow-up 6dfe9da (tests and two doc paragraphs; no src/ or include/ change). Its own message follows verbatim so scripts/check_doc_reverts.py reads the sentences it rewrites or qualifies in docs/design/o1-report-contract.md and docs/pages/run-failure-codes.md. tests, docs: REV's follow-ups on the scratch series: the sink_fd rows name their real subject and cause, the capability row's cells are meaningful and disclosed, the predicate scan covers the signed spellings, the docs state the ENOTSUP exception tests/test_run_json_sink_fd.py: the comment of the unjailed TMPDIR row now says the pre-series failure came from the native writer opening work/bad.sink in the mode-0555 work directory (sink_io), not from TMPDIR; new row test_native_fd_fsize_zero_path_sink_refused_in_read_only_work restores that subject (work 0555, TMPDIR 0755, report_sink_failed sink_io, nothing created); the TMPDIR-cause row runs without an arena, without PINEFORGE_COMMAND_SCRATCH_DIR and PINEFORGE_RUN_JSON_PROFILE, and proves DAC enforcement with an os.open probe first (visible pytest.skip with its reason otherwise). tests/test_default_scratch_effective_credentials.cpp nonroot_row() only (row() and main() unchanged): the setreuid outcome check can fail (EPERM or EINVAL accepted, credentials verified on both branches), the disclosure text is a pure fixture function pinned in test_scratch_capability_parse (every omitted cell "<cell> omitted (<reason>)", "missing, unreadable or malformed"), four more parser cases (0x80, -80, +80, space after the colon). tests/test_scratch_filesystem_predicate.py: the scan and its self-test also cover -2054924042(L) and 0xffffffff858458f6. docs/design/o1-report-contract.md and docs/pages/run-failure-codes.md: one paragraph each states the ENOTSUP exception (no probe, or a real errno 95 from stat/faccessat/statfs, keeps the default candidate as in 1.6.3 with no tmpfs claim and no note; /var/tmp selected without the rejected-candidate note when its probe answers ENOTSUP) and qualifies, with "except as described below", the sentences "A failed filesystem query stops default resolution with its syscall errno and names the failed candidate at every create wrapper; it never permits fallback." and "If creation in the selected `/var/tmp` fails, the detail names that directory, the rejected first candidate and `set TMPDIR to a disk-backed directory`." (o1-report-contract.md) and "A failed query stops resolution with its syscall errno and names the failed candidate." and "failures and all-memory refusals" (run-failure-codes.md); the other sentences keep their words and only their line wrapping changed (same number of lines in each block). No production code changed. Mutations m15 (setreuid EFAULT) and m16 (signed magic). Also in this commit, frozen 2026-10-24 for the 1.7.0 tag (three additions to the tree the CR-150 legs of 6f8f40e measured; src/ and include/ are unchanged): 1. CHANGELOG.md, section "Costs that remain": three new bullets (ORG-CX), 49 lines added and none removed, so no sentence is deleted. They state the costs that remain with the symbols they name: the I/O paths that can terminate or fail silently, the reads of closed trades and cohorts that now cost I/O, and the report-sink cost. 2. tests/test_scratch_arena.cpp (LAB-CX, commit 280c472 on parent 0f5072d, whose tree equals the one measured; test-only, +25/-4): on the CLOUD Batch VM the container filesystem answers open(O_TMPFILE) with errno 95 (EOPNOTSUPP), measured on lab job rj-20261024t121649-2fe44d, so the row failed there with the bare message "create backing descriptor" while it passes on NTU2 (rj-20261024t121045-c354ce). anonymous_regular_backing now falls back to mkstemp + unlink ONLY when the O_TMPFILE open fails with EOPNOTSUPP, so the anonymous regular-file arena shape is still exercised; it prints a NOTE with the errno when it does. Any other errno still fails, now with "errno N (text)" in the message. Nothing is skipped and no assertion is weakened: after the descriptor exists the row runs the same preallocation, attachment, ensure_size and zero-read checks as before. The engine never opens an O_TMPFILE. Docs addition 3 (ORG-CX R9-F1/F2, docs only, line counts unchanged): three public pages stop promising an envelope run_json never emits and a NULL sink removal that can be refused. Its own message follows verbatim so scripts/check_doc_reverts.py reads the nine sentences it replaces. docs: three public pages stop promising an envelope run_json never emits and a NULL sink removal that can be refused REV's read of the frozen tree (R9-F1, R9-F2): the code and the CHANGELOG are right, three pages are not. report-schema.md (lines 253-267) said explicit-mode JSON carries `report_contract`/`report_capture` and that explicit modes add policy keys and move the fingerprint to version 3. On the `run_json.py` transport an explicit `--report-mode FULL` is the legacy route (docker/run_json.py: "--report-mode FULL is identical to omitting the option"; only SUMMARY_ONLY and STREAMED are refused as `report_mode_unsupported`), so it adds none of those keys; they appear on a host that fills the v1 envelope (C++ `fill_report_envelope_v1`). o1-report-contract.md (lines 47-55) made the same claim for every explicit request. abi-stability.md (lines 165-172) listed -1, -2 and -3 for the sink setters but not that NULL returns -3 while an idle handle still holds a completed run's scratch curve or sealed rows (src/engine_report.cpp set_report_sink_v1 and set_report_sink_fd_v1), that an invalid scratch-arena environment returns -1, or that a completed fd generation detaches and keeps its receipt and summary (include/pineforge/report_sink.h documents all three). Line counts of all three pages are unchanged (818, 584, 672), so no citation moves. Sentences replaced (named for the doc-reverts check): - "Explicit-mode JSON includes `report_contract: "pineforge-report/v1"` and `report_capture`, containing `version`, the mode name, `complete`, and all seven sequence objects above." - "Explicit modes also add `report_contract`, `report_mode`, and `sequence_capture` to `applied_runtime` and fingerprinted `provenance.runtime`." - "Either sink setter replaces the one installed sink, and NULL through either setter removes it." - "Both return 0 on acceptance, -1 for a bad handle or descriptor prefix (or a negative fd), -2 during a stream and -3 during a batch run or report capture." - "Run-begin validation and write errors use the existing typed `report_sink_failed` path." - "The caller keeps the fd open through execution and report capture, then closes it to signal EOF." - "The engine never duplicates or closes a borrowed fd, including on failure, replacement or handle destruction." - "No option keeps the legacy route." - "An explicit request must either return the requested versioned envelope or fail." Frozen2 additions (2026-10-24, after REV's P1 on the adapter retirement sweep, R6a-F1): the tree of the first frozen commit cad8b4b plus exactly four changes, nothing else. 1. src/source/pine_adapter.cpp (CR150-CX, commit f5deaf8, parent cad8b4b, 6 insertions and 6 deletions): erase_retired_rows keeps a dormant askable-origin exit leg pinned for its cycle, as v1.6.3's K1 pinned it. The retirement sweep released a dormant askable-origin exit leg A when a later same-family, same-source, same-from_entry exit with different stop bits existed (release_superseded), dropped it from the K2/K3 candidates, and let K3 keep it anyway as the first witness of a live exit Y. A's own supersessor B (released by a later C, nobody's witness, no projection_predecessor link) was then erased. revive_brackets_after_margin decides "superseded" only over retained rows, so with B gone and C's stop bits equal to A's, a margin call restored A as a marketable flatten at the margin price under A's exit id: an extra trade that v1.6.3 never books (v1.6.3 pinned every unwithdrawn lifecycle row for the cycle). held_by_cycle and the matching collection of held legs in the unpinned pre-pass now require the leg's origin NOT to be askable, dormant or not: a dormant askable-origin leg is pinned by pinned_by_itself exactly as in v1.6.3 and is never released. Withdrawn legs, non-askable origins and everything else are unchanged. The cost: a script that keeps re-issuing strategy.exit(qty=) over a dormant askable leg without cancelling retains its rows while the origin stays open, as v1.6.3 did. 2. tests/test_k1_askable_origin_supersessor_erasure.cpp and tests/cr150.d/p2cx-k1.cmake (P2-CX, commit b4c5a42, parent cad8b4b; the registration is the tree's own tests/cr150.d/*.cmake glob, so a clean checkout builds it): the K1 reproduction of REV's R6a-F1 tape runs on two fresh hosts, default ERASE and RETAIN-every-row. Per-bar leg, dormancy and position state is printed for both, controls assert that each tape condition held, and the only oracle is erase trades == retain trades (no literal trade list). A red here (erase books an extra margin-price flatten under "X") proves REV's P1. Green on NTU2 with the fix: job rj-20261024t140803-0bf6e2, 8/8, ERASE == RETAIN. 3. CHANGELOG.md (ORG-CX, patch CHANGELOG-171-DORMANT, sha256 1407fc7f980338d4b11ef31ab25aac5b7df3a9a299e8db09757307b708b0b610): a new last bullet of "Costs that remain", "Re-issued exits on a dormant leg keep one placement row per re-issue" (card ORG-171-DORMANT), and one exception clause in the "placement retirement rules (CR-150)" entry. The sentence it changes (named for scripts/check_doc_reverts.py): "for the dormant exit legs of one origin, exit id, family and entry only the oldest is kept as the representative, with superseded dormant legs released; the retired rows leave the hashed placement sequence, so hash values change wherever such a placement retires;" now reads "..., with superseded dormant legs released (except a dormant leg that was not withdrawn while its origin is still open: it is kept until its position cycle ends, see Costs that remain); the retired rows leave the hashed placement sequence, so hash values change wherever such a placement retires;". 4. The message of the first frozen commit is carried verbatim above, with ONE number corrected: "Line counts of all three pages are unchanged (818, 584, 672), so no citation moves." (the first message said 817; docs/pages/report-schema.md has 818 lines). Frozen3-MIN addition (2026-10-25, CORE-TOP/L6 rulings of 01:51 and 02:14): the tree of the second frozen commit c82fd00 plus exactly the inputs below, nothing else. No source or header file other than the three named in paragraph 5 changes relative to c82fd00. 5. include/pineforge/engine.hpp, include/pineforge/trade_ledger.hpp, src/engine_orders.cpp, tests/cr150.d/p2cx-tledger.cmake, tests/p2cx_tledger_fixture.hpp, tests/test_p2cx_tledger_budget.cpp, tests/test_p2cx_tledger_io.cpp (P2-CX, the cumulative patch of commits 6548775 and d21ea8d on c82fd00, sha256 05b77f191b974e09f028a6b0d9942798ec1a44e20c9de4410866e7e01cd5d3e6, 1181 insertions and 46 deletions; the tests register through the tree's own tests/cr150.d/*.cmake glob): TradeLedger gets a budgeted read cache of decoded closed rows and per-field accessors. A script that reads strategy.closedtrades every bar decoded each historical row per field call (an index pread below the 64-row descriptor window and four string preads outside the 10752-byte payload tail) and every Pine accessor built a whole Trade by value to read one field. Now TradeLedger::<field>_at(i) returns one field straight from the cached record (numeric fields build no Trade and no string; string fields read only the requested packed bytes), with the same checked-row contract as at(); the closed_trade_* accessors in engine.hpp use them, with values, error behaviour and percent formulas unchanged. The cache is direct-mapped by global index; the default budget is 65,536 rows (maximum 1,048,576); one entry is a 168-byte record plus 16 bytes of key and generation plus 128 inline string bytes, 312 bytes (20,447,232 bytes at the default); a row whose four strings exceed 128 bytes keeps its numeric fields cached and reads strings from scratch; the array grows lazily to min(budget, next power of two >= max(64, rows)), so memory is bounded by the budget independent of bars. seal() no longer invalidates the cache (publication changes no row content and the checked accessors refuse sealed rows before reaching the cache); replace, rewind (resize), move and clear invalidate it; replace() reuses an unchanged string extent instead of appending the same bytes again. kResidentRows (64), kRecordBytes, resident_size() and resident_base() are unchanged. No environment variable or run-spec field controls the budget: the default is what runs (C++ API only). The budget setter's refusal still uses setting_rejected until the shared resident-budget code lands. Tests: test_trade_ledger, test_p2cx_tledger_io (syscall counter, zero reads and writes in a hungpixi-shaped 24,238-row loop after warm-up, oracle-compared) and test_p2cx_tledger_budget (budget formula, lazy growth boundaries, wide-string fallback, replace, rewind and move invalidation, sealed-row refusal for every accessor); P2-CX reports them Passed on NTU2 (job rj-20261024t175023-a20f66) with mutation rows red on the intended assertions. SM-CX measured this patch on frozen2: hungpixi 106.14 to 107.58 s over four repetitions against 72.84 to 73.55 s on 1.6.3, trades identical in every cell; the engine still makes scratch reads and writes per bar on other paths (id tables, exit scratch, journal), which is the cost named in the CHANGELOG. 6. CHANGELOG.md and documentation pages (ORG-CX; patches CHANGELOG-171-DORMANT-V2, OVERCLAIM-170-v4, REPIN-FROZEN3MIN-NOPAGE, CHANGELOG-170-SLOWER-FILLED-v3 and CHANGELOG-LEDGERCACHE-v2, applied in that order after paragraph 5; CHANGELOG-LEDGERCACHE-v2 rewrites the opening of the "Reads of closed trades and cohorts now cost I/O" bullet because the closed-trade read cache of paragraph 5 makes its statement about scratch reads on every call false: it now states the cache, its default budget of 65,536 rows of 312 bytes (20,447,232 bytes at most per engine), its lazy allocation at the first closed trade of every Pine run, that above 65,536 closed trades it is direct-mapped so a loop over the whole history re-reads colliding rows from scratch files on every pass, and that no user control sets it in this release): the dormant-leg bullet of "Costs that remain" is rewritten (V2), nine over-broad memory sentences are corrected (v4), the engine.hpp citation numbers in docs/adr/0001-kernel-adapter-boundary.md, docs/design/native-feature-parity.md, docs/pages/native-engine.md and docs/pages/pine-to-native.md are re-pinned by the anchor tool because paragraph 5 moves engine.hpp lines by one or two (the box anchor check reports every documentation anchor resolved), and a new first bullet of "Costs that remain" states that 1.7.0 is slower than 1.6.3 on some scripts with the figures SM-CX measured (no existing sentence is replaced by that bullet). No memory-management page is added. The sentences replaced or deleted by this paragraph, named for scripts/check_doc_reverts.py: - "The bounded paths of this release are the trade ledger, the report sink and the command journals described above." (CHANGELOG.md, Costs that remain) - "The complexity in the number of rows is unchanged; the cost per row is not." (CHANGELOG.md, Costs that remain) - "Origin/cohort receipts, bracket rosters, historical IDs, successor roots and position tails use retirement, bounded resident windows or scratch instead of retaining all history in RAM." (CHANGELOG.md, Adapter history retirement) - "the production Pine host uses KernelRecordedAtHostMarks (src/source/pine_adapter.cpp:2870), whose recorded curve is held in O(1) and streamed to the scratch file." (docs/design/cr150-contracts.md) - "Complete enabled sequences, in existing report order, through a bounded sink" (docs/pages/report-schema.md, STREAMED row) - "Fixed slab for historical table bytes, prepared before the run, with count/byte admission and no growth or disk fallback on exhaustion." (docs/pages/scratch-tables.md, bounded_memory row) - "For the short waiting-child parent-terminal fixture, the book's 4,096-row cap and the journal's 2,048-event cap give a per-engine definition plateau of 1,890,980 bytes (about 1.9 MB) on GNU 13.4 x86-64, independent of symbols, window and trades." (docs/pages/scratch-tables.md) - "caps (book 4,096 + journal 2,048 events), constant, independent of symbols, window and trades." (docs/scratch-store.md, resident bound of the parent-terminal drain) - "for the dormant exit legs of one origin, exit id, family and entry only the oldest is kept as the representative, with superseded dormant legs released; the retired rows leave the hashed placement sequence, so hash values change wherever such a placement retires;" (CHANGELOG.md) - "**Reads of closed trades and cohorts now cost I/O.** The prefix sums described above apply to sealed rows only. Unless history is streamed, the trade ledger keeps 64 rows resident and reads older closed rows, text included, back from scratch files on every call, where the previous release walked an in-memory vector (`std::vector<Trade>`)." (CHANGELOG.md, Costs that remain, closed-trade reads bullet; rewritten by CHANGELOG-LEDGERCACHE-v2) - "`BacktestEngine::avg_trade_percent`, `avg_winning_trade_percent`, `avg_losing_trade_percent` and the `closed_trade_*` accessors therefore read every older row per call, so a script that calls them each bar pays one scratch read per row per call." (CHANGELOG.md, same bullet; rewritten by CHANGELOG-LEDGERCACHE-v2) Every citation line changed by the anchor re-pin differs from main only in a line number inside the same sentence; the doc-reverts checker normalises numbers, so none of them is named above. The message of the second frozen commit is carried verbatim above (frozen2 paragraphs 1 to 4, which in turn carry the first frozen commit's message). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The test replaces the global operator new/delete with malloc/free to count allocations, but not the nothrow forms. CommandScratchStore allocates with `new (std::nothrow)`, so under ASan that block came from the sanitizer's own nothrow new and was released by the replaced operator delete's free(): alloc-dealloc-mismatch abort (sanitizers leg of PR #496, rj-20261024t184800-7cdb3a, test_p2cx_tledger_io). Add malloc-backed operator new/new[] (std::nothrow_t) with the same counter, and the matching nothrow operator delete/delete[]. Test only; no src or include change. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
PR #496's kernel leg failed test_selected_metrics_l17b (sink heap-flatness witness, 12 checks): the ledger read cache was allocated by the consumer's own closed-row reads and grew with the closed-trade count to 65,536 x 312 = 20,447,232 B in sink runs, against a budget of 2 x kScratchWindowBytes with large <= small + window. Owner rule: the witness is not loosened. At run begin the consumer now sets the ledger cache budget from the run mode: TradeLedger::kSinkCacheRows = 128 rows (128 x 312 = 39,936 B, under one 64 KiB scratch window, independent of the trade count) when a report sink is installed, the default 65,536 rows otherwise (sink-off runs, which is how the verifier and the sweep run, are unchanged). set_cache_rows is called on a freshly cleared ledger, so it cannot hit the frozen refusal, and it runs every begin so a reused handle never keeps a stale sink budget. Arithmetic against the witness: other working storage 65,592..65,928 B (window plus 56..392) + 39,936 = at most 105,864 <= 131,072 (2 windows); small (1k trades) and large (100k trades) both allocate exactly 128 x 312, so large - small = 0 <= window. A static_assert in test_p2cx_tledger_budget pins kSinkCacheRows x kLedgerEntryBytes <= 64 KiB. Mutations: the sink branch using the default budget (l17b red); every run using the sink budget (the hungpixi-shaped I/O test red). Based on ef4c628 (nothrow operator new fix). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…n a report sink is installed (LEDGERCACHE-v3) Documents P2-CX's sink-on bound (commit 38e00f3): NativeExecutionConsumer::begin_ready sets the ledger read-cache budget at every begin, 128 rows with a report sink installed (report_sink_state_.enabled) and 65,536 rows otherwise; a cache entry is 312 bytes, so the cache is at most 39,936 bytes with a sink and 20,447,232 bytes without, allocated at the smaller of the budget and the next power of two at or above the number of closed rows (at least 64). The direct-mapped miss cost is stated for both budgets, and the budget has no user control. Applied with ORG-CX's patch CHANGELOG-LEDGERCACHE-v3-DELTA-on-69467f7cc2 (sha256 5841f8c3bd9e3993e04cea7038de38eaefd632c7d358276da3046babefc04ea9), CHANGELOG.md only. Sentences replaced in CHANGELOG.md (Costs that remain, the closed-trade cache bullet; they are from this PR's own earlier text, not from main), named for scripts/check_doc_reverts.py: - "The cache holds at most `TradeLedger::kDefaultCacheRows` = 65,536 rows of `TradeLedger::kLedgerEntryBytes` = 312 bytes, 20,447,232 bytes, and is allocated at the smaller of that budget and the next power of two at or above the number of closed rows (at least 64): 19,968 bytes up to 64 closed trades, 10,223,616 bytes at 24,238, 20,447,232 bytes above 32,768." - "It stays until the next run begins or the engine is destroyed, and each engine holds its own, so K engines alive together hold K times that." - "Above 65,536 closed trades the cache is direct-mapped, so rows that share a slot evict each other and a loop over the whole history reads them from scratch files again on every pass: 2 x (N - 65,536) rows per pass for N closed trades below 131,072, and every row from 131,072 closed trades on." - "The budget has no user control in this release: `TradeLedger::set_cache_rows` is a C++ call that nothing in the engine, the C API or the environment makes." Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…nd shifts (anchor tool, ORG-CX) The sink bound (commit 38e00f3) adds 5 lines to src/native_execution_consumer.cpp near line 3168 and 3 lines to include/pineforge/trade_ledger.hpp, so every citation of native_execution_consumer.cpp:N with N above 3168 moves by exactly +5. Applied with ORG-CX's patch REPIN-FINAL6CAND-7968dbf (sha256 022af9f15a88e1304df68c34b31708b998100c39a0c0321d86816ef8cfdb21ff): 7 documentation files, 79 insertions and 79 deletions, each changed line differs from its original only in digits (the digit-masked check ONLY_NUMBERS_CHANGED holds for all 79 lines), so no sentence is added, removed or reworded; the box anchor tool reports 2209 anchors, 2209 ok, 0 bad on the patched tree. Files: docs/adr/0001-kernel-adapter-boundary.md, docs/design/arena-bound.md, docs/design/cr150-contracts.md, docs/design/native-feature-parity.md, docs/native-refactor-progress.md, docs/pages/native-engine.md, docs/pages/pine-to-native.md. Sentences replaced for scripts/check_doc_reverts.py: none (citation numbers only). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
This was referenced Oct 24, 2026
Closed
Closed
Closed
luisleo526
added a commit
that referenced
this pull request
Oct 24, 2026
…pointers, macOS build fix (::sig*set macros) (#500) Three commits, squashed; each message follows verbatim so scripts/check_doc_reverts.py reads the sentences they name. CHANGELOG: say which entry points run with a report sink, and what that costs on the release image The closed-trade cache bullet said how big the ledger read cache is with and without a report sink but not which entry points install one. It now names them, from the tree: a handle has no sink until its host calls strategy_set_report_sink_v1 or strategy_set_report_sink_fd_v1; the C API used directly, C++ hosts including the Pine host, the runner/ command line, scripts/run_strategy.py without PINEFORGE_REPORT_SINK=1 and docker/run_json.py --profile jailed without --report-sink-fd keep the 65,536-row cache (20,447,232 bytes); docker/run_json.py in its default unjailed profile (it installs a temporary sink on every handle), the jailed profile with --report-sink-fd, run_strategy.py with PINEFORGE_REPORT_SINK=1 and any host that installs a sink get the 128-row cache (39,936 bytes), and a closed-trade loop over more than 128 rows re-reads the colliding rows from scratch files on every pass (every row from 256 closed trades on). The App's jailed host is stated as not known. The "Costs that remain" slowdown bullet gains the measurements that go with it, each taken on one 16 vCPU machine: the six-script figures were measured through scripts/run_strategy.py, which installs no sink unless PINEFORGE_REPORT_SINK=1 (it was not), so the large cache was in effect; hungpixi (24,238 closed trades) through the release image's default docker/run_json.py took 74.18 s on 1.6.3 and did not finish within 1800 s on 1.7.0 (the run was stopped at that limit); through scripts/run_strategy.py without a sink it takes 107.2 s (mean of three runs); peak RSS of the default run_strategy.py path (hl2-01, 100,000 bars, n=3) is 144.9 MiB on 1.6.3 and 125.7 MiB on 1.7.0. No statement is made that the run_json.py results are identical: the 1.7.0 run produced none. Replaced sentences (all four are in the closed-trade cache bullet of CHANGELOG.md on main 91353de, the 1.7.0 line squash of #499, and are replaced here): - "Without a report sink the cache holds at most `TradeLedger::kDefaultCacheRows` = 65,536 rows of `TradeLedger::kLedgerEntryBytes` = 312 bytes, 20,447,232 bytes, and is allocated at the smaller of that budget ... 20,447,232 bytes above 32,768." is now the sentence beginning "A run with no report sink holds at most". - "With a report sink installed the budget is `TradeLedger::kSinkCacheRows` = 128 rows, 39,936 bytes, a fixed bound ... so that the sink run's working memory stays flat." is now the sentence beginning "A run with a report sink installed holds at most". - "Above the budget the cache is direct-mapped, so rows that share a slot evict each other ... with a report sink, 2 x (N - 128) rows per pass below 256 closed trades, and every row from 256 on." keeps both formulas and now says "with no sink" and "with a sink". - "The budget has no user control in this release: `NativeExecutionConsumer::begin_ready` sets it at every run begin ... and nothing in the C API or the environment changes either figure." is now "The two figures cannot be changed by a user in this release: ... installing a sink or not is the only choice." (the environment variable PINEFORGE_REPORT_SINK does choose whether run_strategy.py installs a sink, so the old sentence overstated). One more sentence of main's text is reworded in the same bullet: "The six are not a sample of all scripts, and the results are identical in every measured case." now ends "identical in every one of those six measured cases", because the run_json.py hungpixi run added below has no 1.7.0 result to compare. The not-affected sentence names the hosted case runner, the verifier and the sweep, and states that the corpus-parity canary pass sets PINEFORGE_REPORT_SINK=1 (scripts/check_corpus_parity.sh, line 316). The heading of the 1.7.0 section is renamed: the line "## Unreleased" is replaced by "## 1.7.0 — 2026-10-25" (same form as "## 1.6.1 — 2026-10-18"). VERSION is not touched here. Only CHANGELOG.md changes; no citation moves. docs: drop the stale "unreleased" statements now that the 1.7.0 section carries its release heading The CHANGELOG heading "## Unreleased" became "## 1.7.0 — 2026-10-25" in the previous commit. Three public statements still said the change was unreleased or pointed at the old heading. Each is changed by words only; no line is added or removed, so no file:line citation moves. Replaced text (sentence or line as it stands on the previous commit): - docs/pages/public-contract.md, "From 1.6.2 (unreleased), the image declares selected-window contract" is now "From 1.6.2, the image declares selected-window contract" (the 1.6.2 and 1.6.3 sources already declared revision 1.3, as README.md:61 and docs/pages/report-schema.md say). - docs/pages/public-contract.md, "[CHANGELOG.md](../../CHANGELOG.md), under *Unreleased*, inventories this change." is now "[CHANGELOG.md](../../CHANGELOG.md), under *1.7.0*, inventories this change." (the heading it points at no longer exists). - docker/README.md, "From 1.6.2 (unreleased), the engine/codegen image's `run_json.py --so <strategy.so> --capabilities-json` probe declares revision `1.3`" is now "From 1.6.2, the engine/codegen image's `run_json.py --so <strategy.so> --capabilities-json` probe declares revision `1.3`". Left as they are, from `git grep -in unreleased` over the tree minus CHANGELOG.md and docs/build.sh: the schema field `"unreleased": true` of the run-failure catalog diff (docker/run_failure_codes_diff.json, docs/pages/run-failure-codes.md, scripts/gen_run_failure_catalog_diff.py), the README.md:715 note on the documentation version label, and the "unreleased epoch" comments in tests/ that record why pinned hashes moved; none of them says that this release is unreleased. Build on macOS: call sigemptyset and sigaddset without the global-scope operator On macOS signal.h defines sigemptyset, sigaddset, sigdelset, sigismember and sigfillset as function-like macros (MacOSX.sdk usr/include/signal.h:122-126, for example `#define sigemptyset(set) (*(set) = 0, 0)`). A call written `::sigemptyset(&mask)` expands to `::(*(&mask) = 0, 0)`, which clang rejects with "expected unqualified-id", so src/c_abi.cpp and src/scratch_arena.cpp did not compile there (found by the release.yml dry run of the prebuilt macos-universal job and by the macos-26 Debug and Release legs of ci-post). Linux declares them as functions, so the `::` form compiled everywhere else and pull request CI never builds macOS. The `::` is dropped on every call of the five names in src/, include/ and tests/: 14 call sites in 10 files, each an in-line edit of one token (12 lines changed, no line added or removed, so no file:line citation moves). The unqualified name finds the function on Linux and the macro on macOS. src/c_abi.cpp:368, src/scratch_arena.cpp:369, tests/report_sink_failure_fixture.hpp:78, tests/test_command_scratch_arena_bounded.cpp:647, tests/test_o1bk_production_backing.cpp:716, tests/test_p2_report_abort_security_diag.cpp:16 (two calls), tests/test_report_scratch_arena.cpp:467, tests/test_run_sigterm_abort.cpp:23 and :24, tests/test_scratch_arena.cpp:1156 and :1183, tests/test_scratch_arena_jailed_seals.cpp:114. Class check, over src/, include/ and tests/: a grep for a global-scope `::name` of the other macros macOS headers define (FD_SET, FD_ZERO, FD_ISSET, FD_CLR, htons, ntohs, htonl, ntohl, major, minor, makedev, the W* wait macros, errno, isnan, isinf, isfinite, getc, putc, getchar, putchar, feof, ferror, clearerr, stdin, stdout, stderr, the ctype functions, S_IS*, CMSG_*, memcpy, memmove, memset, strcpy, snprintf, sprintf, bzero, timeradd and its relatives, setjmp, longjmp, assert, offsetof, NULL, va_*) finds none. The only other global-scope names that the macOS SDK could define as macros are ::dirfd (a statement-expression macro in dirent.h, but only when the deployment target is older than macOS 10.8) and ::fileno (not a macro in the SDK read here); both are left unchanged and are named in the report to the supervisor. Only source and test files change; no documentation text is replaced. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine 1.7.0 release (frozen3-MIN, final6). Supersedes #498 (4ae6b04) with three commits on its head:
Evidence: #496 sweep 134/134 (hungpixi excellent 2423/2423), parity dry run PASS, CR-271 identity 312/0 (sweep and identity run sink OFF; the sink bound does not change sink-off code paths); #498 preflight + sanitizers green (ASan test fix). All 8 steps + private release on THIS tree ce1311be7bb1 (rj-20261024t195707-99513f ... ); a new gate sweep + CR-271 identity on this head follow. REV-CX GO on 38e00f3 (bound in bytes and the sink-off reason checked against the diff).
🤖 Generated with Claude Code