report sink: retire sealed closed-trade rows on sink routes when the module opts in by luisleo526 · Pull Request #591 · pineforge-4pass/pineforge-engine

luisleo526 · 2026-10-27T23:42:30Z

Replaces #583 as one commit on main (same product code; the commit message names every replaced memory-page sentence).

What changes

Eligible report-sink runs release closed-trade descriptors and text after the existing readers have consumed the sealed prefix. Indexed reads into that released prefix refuse explicitly; runs that retain history continue to return the original rows.

Retirement requires all of: an installed path or borrowed-fd sink, set_trade_history_streamable(true), and an owning strategy module whose existing generated export strategy_reads_trade_history() returns zero. The codegen already emits that export and pins it across every closed-trade read shape (codegen #213, #264), so no new symbol and no compiler change are needed. A missing symbol, a nonzero result, a failed lookup, an exception, or unproved module ownership retains full history. The lookup is module-local and image-checked (the symbol must resolve in the same image as the strategy object).

The existing seal boundary stays authoritative. Trade records, identity strings, report/metric/money folds and hashes are consumed before release; a failed write advances no retirement base and publishes no trailer. Below-base closed_trade, get_trade, previous_closed_trade, Pine field reads and C metadata reads on admitted retiring runs carry the existing engine_invariant with empty arguments {}. The C metadata accessors keep their existing return sentinels.

What does NOT change

No-sink runs retain their complete owning history, even when the module exports zero. Nonstreamable scripts and modules that export nonzero or nothing retain historical indexed reads. Logical indices, seal predicates, mutable exit-time groups, report bytes, numeric folds, broker/source hash recipes, installed layouts and the runtime failure-code and export sets are unchanged. Physical storage origins are excluded from logical hashes.

The byte-identity proof compares complete sink bytes, metrics, every per-bar source/broker hash and the final hashes with a frozen reference produced by the unchanged engine (see "How to verify"); the reference is committed under tests/data/trade_history_reference/ (15 files, 4,716,341 bytes in total) with a SHA-256 manifest, so the check runs in ordinary CI.

Tests

Test	What it asserts
`test_report_sink_trade_retirement`	Real Pine fills and closes at 4,000 / 16,000 / 64,000 trades with a terminal open row, path and socket sinks, disk and arena backing, short and 513-byte text. Descriptor and payload occupancy are sampled from the actual backing and must stay flat; the refusal subrow checks owning, Pine and C reads, an unchanged backward cursor, and the retained suffix after copy and move.
`test_report_sink_trade_history`	An explicit nonzero export retains growing backing and exact oldest-row reads on later callbacks.
`test_report_sink_trade_legacy`	A missing export retains the same history and owning reads.
`test_report_sink_trade_identity`	An independent wire parser checks trade populations, numbering, model fields, receipt counts and the digest; the bytes, metrics and hashes must equal the frozen unchanged-engine reference.
`test_report_sink_trade_reference`	Flips one byte in each of the 14 manifested reference files: every flip must be rejected; a missing or empty reference directory still fails the identity row.
`test_sink_history_consumers`	Product-host path and fd assembly and the strategy-runner CSV flow succeed with the metadata history accessors replaced by raising functions; no historical accessor is called.
`test_report_sink_trade_residency`	Existing heap, window, path and fd assertions, with the native tape corrected to include the required fill bar.

Result of the six rows above on the pushed history before the final rename commit: all passed on a disk-backed lab VM (retirement 204 s, history 191 s, legacy 189 s, identity 13.5 s, reference 0.17 s, consumers 0.8 s). The rows, the six verification legs and the parity sweep are re-run on the head of this PR.

Stated limits

This releases only the retired ledger prefix. Live unsealed rows, a pinned exit-time group and its required predecessor remain live. Equity scratch, exit phases, position-tail spill, native driver, account and command histories, and the historical cohort, id-pair, rank and successor stores have separate lifetimes and are not changed here. Whole-arena usage is not established by ledger-extent checks.
Full-history fallback stays linear in closed trades: a 168-byte descriptor plus variable strings per row. The no-sink FULL projection and report contract remain history-sized.
The retirement witness bounds the two ledger extents (per-extent peak at most 2 MiB + 64 KiB; spread across populations at most 1 MiB + 64 KiB); these are test bounds for the extents, not a whole-run memory cost.
Release-image whole-arena results on path and borrowed-fd sink routes at 100k, 300k and 1M bars are owed and measured before merge, not claimed here.
The app converter was not inspected for this change. The in-repo consumer checks do not establish every external consumer's compatibility; the application team's evidence that no consumer reads below the published base is tracked separately.
Real generated false, true and missing-export modules from the current codegen, and multiple-module isolation, are exercised by the module-image checks in the retirement and legacy executables with locally defined exports; an end-to-end run with a codegen-built module on the release image is owed before merge.

How to verify

Frozen reference: scripts/regenerate_trade_history_reference.py --base-tree <clean worktree of the commit in tests/data/trade_history_reference/BASE_COMMIT> builds the legacy executable only from that base tree in a fresh harness, writes the artifacts and the manifest, and refuses to publish more than 5,000,000 bytes. It never uses the changed library.
Rows: ctest -R '^test_report_sink_trade_(identity|reference|retirement|history|legacy|residency)$|^test_sink_history_consumers$' on a disk-backed runner.
Gates: python3 scripts/ci_preflight.py (documentation anchors, source inventory), clean full Release verification with scripts/ci_verify.py, and the corpus parity sweep; all report 0 differences for a change that must not alter any report byte.

🤖 Generated with Claude Code

…module's export allows it With a report sink installed, a streamable policy and a strategy module whose existing export strategy_reads_trade_history() returns zero, closed-trade descriptors and text below the published base are released after the existing readers have consumed them; a read below the base refuses with the existing engine_invariant (empty arguments). Every other run keeps its complete history and the no-sink contract is unchanged. Report bytes, folds and hashes do not change. - The identity row compares the candidate with a frozen reference produced by the unchanged engine and committed under tests/data/trade_history_reference (generated by scripts/regenerate_trade_history_reference.py at the pinned base, SHA-256 manifest verified before use, 4.7 MB). The comparison normalizes only the engine string of the sink header and the trailer digest that covers it, so a release version bump needs no new data. - strategy_history_policy.hpp uses dladdr/dlopen/dlsym, so both archives link the dynamic loader library (CMAKE_DL_LIBS). - The new trade rows are slow rows and are added to the pinned slow set. - Documentation anchors into c_abi.cpp, scratch_file.hpp and CMakeLists.txt are re-pinned after the line shifts of this change. The memory page text that this change reworded or removed is named here for the documentation revert check: - A sink moves report rows to its path or fd stream while the ledger's external scratch still needs a writable, disk-backed backing store. - Older rows and all string payloads: 168 B plus text per closed trade - Install a report sink; the sink route is bounded - Adapter state (placement rows, settled bracket families, script containers) is separate and has its own rows below - Heap bounded for the report; adapter state as the rows below - A default run with no report sink needs a writable, disk-backed scratch directory even when it makes no trade. - The spilled trade rows are `O(T)` on disk with a small resident window (see below). - C or C++ host with no sink, and jailed Python without `--report-sink-fd` (the default route of the C API): equity curve - Install a report sink (above); a bounded default route is planned, with no release date - Same route (no sink): closed trades - `T`: 168 B plus text per trade - Scratch, with 64 descriptor rows resident and a read cache of up to 65,536 entries (312 B each, allocated as trades accumulate); the FULL report built - Install a report sink; no release date for the default route - Any report sink installed (native or Pine host, history streamed or not) - The heap is bounded for the equity curve (one held point and one 64 KiB batch) and for the ledger (64 descriptor rows and a 128-entry read cache). - The trailing block of rows that share the last exit time stays resident only when history is streamed (`K` lots closed at one time, `O(K)`). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

+import json
+from pathlib import Path
+import re
+import unittest


+from pathlib import Path
+import shutil
+import tempfile
+import unittest


…aling, UDT graphs, metrics replay and the CI admission fix (#616) This lands twelve engine pull requests as one change, verified and measured once on the combined tree: - #593 scratch: a fixed write-through read cache for the adapter's id tables - #594 live runner: a delta report stream by default, and report-path work that does not rescan closed trades - #597 live runner: typed pre-growth stop for the ledger budget and flat pending counts - #608 run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report - #586 Pine UDT object graphs on the paged object Store - #578 native order: dormant working book, flat per-point cost for never-filled orders - #579 adapter: skip roster passes and empty-family scans that cannot change anything - #599 adapter: read projected pending orders through one cached duplicate index - #563 adapter: gate the reversal exit cancel on candidate owners instead of every historical id - #591 report sink: retire sealed closed-trade rows on sink routes when the module opts in - #596 metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay - #615 ci: accept release.yml's matrix runner under its fetched path in PR admission Fixes found while testing the combined tree are in the final commits: the retiring ledger's cache sizing, a residency witness that bounds sink-run heap by the declared cache cap and checks zero growth at 100k, 200k and 1M trades, a test fix for a descriptor reuse race, and a standing-orders timing grid whose 20,000-order point is tracked in #621. The member pull requests are closed as landed here.

luisleo526 · 2026-10-28T15:57:18Z

Landed in #616 (squash a991b4e on main), which carries this change with one verify and one parity run on the combined tree.

luisleo526 mentioned this pull request Oct 27, 2026

report sink: retire sealed closed-trade rows on sink routes when the module opts in #583

Closed

github-code-quality Bot found potential problems Oct 27, 2026

View reviewed changes

Comment thread tests/test_report_sink_trade_comparison.py

import json

from pathlib import Path

import re

import unittest

Comment thread tests/test_report_sink_trade_reference.py

from pathlib import Path

import shutil

import tempfile

import unittest

luisleo526 mentioned this pull request Oct 28, 2026

Combined 1.8.0 batch: scratch caches, live-runner reports, adapter scaling, UDT graphs, metrics replay and the CI admission fix #616

Merged

luisleo526 closed this Oct 28, 2026

Navigation Menu

Repository navigation

report sink: retire sealed closed-trade rows on sink routes when the module opts in - #591

report sink: retire sealed closed-trade rows on sink routes when the module opts in#591
luisleo526 wants to merge 1 commit into
mainfrom
report-sink/trade-retirement-v2

luisleo526 commented Oct 27, 2026

Uh oh!

luisleo526 commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

luisleo526 commented Oct 27, 2026

What changes

What does NOT change

Tests

Stated limits

How to verify

Uh oh!

luisleo526 commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant