report sink: retire sealed closed-trade rows on sink routes when the module opts in by luisleo526 · Pull Request #583 · pineforge-4pass/pineforge-engine

luisleo526 · 2026-10-27T22:25:36Z

What changes

Eligible report-sink runs release closed-trade descriptors and text after the existing readers have consumed the sealed prefix. Indexed reads into that released prefix refuse explicitly; runs that retain history continue to return the original rows.

Retirement requires all of: an installed path or borrowed-fd sink, set_trade_history_streamable(true), and an owning strategy module whose existing generated export strategy_reads_trade_history() returns zero. The codegen already emits that export and pins it across every closed-trade read shape (codegen #213, #264), so no new symbol and no compiler change are needed. A missing symbol, a nonzero result, a failed lookup, an exception, or unproved module ownership retains full history. The lookup is module-local and image-checked (the symbol must resolve in the same image as the strategy object).

The existing seal boundary stays authoritative. Trade records, identity strings, report/metric/money folds and hashes are consumed before release; a failed write advances no retirement base and publishes no trailer. Below-base closed_trade, get_trade, previous_closed_trade, Pine field reads and C metadata reads on admitted retiring runs carry the existing engine_invariant with empty arguments {}. The C metadata accessors keep their existing return sentinels.

What does NOT change

No-sink runs retain their complete owning history, even when the module exports zero. Nonstreamable scripts and modules that export nonzero or nothing retain historical indexed reads. Logical indices, seal predicates, mutable exit-time groups, report bytes, numeric folds, broker/source hash recipes, installed layouts and the runtime failure-code and export sets are unchanged. Physical storage origins are excluded from logical hashes.

The byte-identity proof compares complete sink bytes, metrics, every per-bar source/broker hash and the final hashes with a frozen reference produced by the unchanged engine (see "How to verify"); the reference is committed under tests/data/trade_history_reference/ (15 files, 4,716,341 bytes in total) with a SHA-256 manifest, so the check runs in ordinary CI.

Tests

Test	What it asserts
`test_report_sink_trade_retirement`	Real Pine fills and closes at 4,000 / 16,000 / 64,000 trades with a terminal open row, path and socket sinks, disk and arena backing, short and 513-byte text. Descriptor and payload occupancy are sampled from the actual backing and must stay flat; the refusal subrow checks owning, Pine and C reads, an unchanged backward cursor, and the retained suffix after copy and move.
`test_report_sink_trade_history`	An explicit nonzero export retains growing backing and exact oldest-row reads on later callbacks.
`test_report_sink_trade_legacy`	A missing export retains the same history and owning reads.
`test_report_sink_trade_identity`	An independent wire parser checks trade populations, numbering, model fields, receipt counts and the digest; the bytes, metrics and hashes must equal the frozen unchanged-engine reference.
`test_report_sink_trade_reference`	Flips one byte in each of the 14 manifested reference files: every flip must be rejected; a missing or empty reference directory still fails the identity row.
`test_sink_history_consumers`	Product-host path and fd assembly and the strategy-runner CSV flow succeed with the metadata history accessors replaced by raising functions; no historical accessor is called.
`test_report_sink_trade_residency`	Existing heap, window, path and fd assertions, with the native tape corrected to include the required fill bar.

Result of the six rows above on the pushed history before the final rename commit: all passed on a disk-backed lab VM (retirement 204 s, history 191 s, legacy 189 s, identity 13.5 s, reference 0.17 s, consumers 0.8 s). The rows, the six verification legs and the parity sweep are re-run on the head of this PR.

Stated limits

This releases only the retired ledger prefix. Live unsealed rows, a pinned exit-time group and its required predecessor remain live. Equity scratch, exit phases, position-tail spill, native driver, account and command histories, and the historical cohort, id-pair, rank and successor stores have separate lifetimes and are not changed here. Whole-arena usage is not established by ledger-extent checks.
Full-history fallback stays linear in closed trades: a 168-byte descriptor plus variable strings per row. The no-sink FULL projection and report contract remain history-sized.
The retirement witness bounds the two ledger extents (per-extent peak at most 2 MiB + 64 KiB; spread across populations at most 1 MiB + 64 KiB); these are test bounds for the extents, not a whole-run memory cost.
Release-image whole-arena results on path and borrowed-fd sink routes at 100k, 300k and 1M bars are owed and measured before merge, not claimed here.
The app converter was not inspected for this change. The in-repo consumer checks do not establish every external consumer's compatibility; the application team's evidence that no consumer reads below the published base is tracked separately.
Real generated false, true and missing-export modules from the current codegen, and multiple-module isolation, are exercised by the module-image checks in the retirement and legacy executables with locally defined exports; an end-to-end run with a codegen-built module on the release image is owed before merge.

How to verify

Frozen reference: scripts/regenerate_trade_history_reference.py --base-tree <clean worktree of the commit in tests/data/trade_history_reference/BASE_COMMIT> builds the legacy executable only from that base tree in a fresh harness, writes the artifacts and the manifest, and refuses to publish more than 5,000,000 bytes. It never uses the changed library.
Rows: ctest -R '^test_report_sink_trade_(identity|reference|retirement|history|legacy|residency)$|^test_sink_history_consumers$' on a disk-backed runner.
Gates: python3 scripts/ci_preflight.py (documentation anchors, source inventory), clean full Release verification with scripts/ci_verify.py, and the corpus parity sweep; all report 0 differences for a change that must not alter any report byte.

🤖 Generated with Claude Code

…: trade residency by route Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… and the module opts in; typed refusal below the base; tests Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ion script and manifest verification (data not yet generated) Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…base Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…port instead of a new symbol Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

+from pathlib import Path
+import shutil
+import tempfile
+import unittest


… pin the real base; link the loader library; pin the new slow rows - The identity comparison normalizes exactly the engine string of the sink header (and the trailer digest that covers it) on both sides, so a release version bump no longer needs new frozen data; every other byte still compares exactly. The rule is stated in docs/trade-history-reference.md and covered by test_report_sink_trade_comparison. - The frozen reference is pinned to the base this change was cut on. - strategy_history_policy.hpp uses dladdr/dlopen/dlsym, so both archives now link the dynamic loader library (CMAKE_DL_LIBS). - The new trade rows are slow rows and are added to the pinned slow set. - Documentation anchors into c_abi.cpp and scratch_file.hpp are re-pinned after the line shifts of this change. The memory page text that earlier commits of this branch reworded or removed is named here for the documentation revert check: - A sink moves report rows to its path or fd stream while the ledger's external scratch still needs a writable, disk-backed backing store. - Older rows and all string payloads: 168 B plus text per closed trade - Install a report sink; the sink route is bounded - Adapter state (placement rows, settled bracket families, script containers) is separate and has its own rows below - Heap bounded for the report; adapter state as the rows below - A default run with no report sink needs a writable, disk-backed scratch directory even when it makes no trade. - The spilled trade rows are `O(T)` on disk with a small resident window (see below). - C or C++ host with no sink, and jailed Python without `--report-sink-fd` (the default route of the C API): equity curve - Install a report sink (above); a bounded default route is planned, with no release date - Same route (no sink): closed trades - `T`: 168 B plus text per trade - Scratch, with 64 descriptor rows resident and a read cache of up to 65,536 entries (312 B each, allocated as trades accumulate); the FULL report built - Install a report sink; no release date for the default route - Any report sink installed (native or Pine host, history streamed or not) - The heap is bounded for the equity curve (one held point and one 64 KiB batch) and for the ledger (64 descriptor rows and a 128-entry read cache). - The trailing block of rows that share the last exit time stays resident only when history is streamed (`K` lots closed at one time, `O(K)`).

The twelve artifacts are byte-identical to the previous freeze; only the base pin and the manifest line for it changed. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

+import json
+from pathlib import Path
+import re
+import unittest


luisleo526 · 2026-10-27T23:42:35Z

Superseded by #591 (same change as one commit).

perfcx and others added 8 commits October 27, 2026 23:26

tests: closed-trade residency witness for the sink route; memory page…

88c7c9c

…: trade residency by route Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

report sink: retire sealed closed-trade rows when a sink is installed…

2a669e1

… and the module opts in; typed refusal below the base; tests Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

tests: use the fixture's shapes parameter in the retirement run helper

e124e31

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

tests: sink CSV row count is two rows per trade side

2ce0f53

tests: frozen trade-history reference with a pinned base, a regenerat…

1ab07a0

…ion script and manifest verification (data not yet generated) Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

tests: frozen trade-history reference data generated from the pinned …

6eb0775

…base Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Merge main (95f4fd1) into the trade-retirement branch

74ea77c

report sink: read the compiled module's existing closed-trade-rows ex…

7e49859

…port instead of a new symbol Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

github-code-quality Bot found potential problems Oct 27, 2026

View reviewed changes

Comment thread tests/test_report_sink_trade_reference.py

from pathlib import Path

import shutil

import tempfile

import unittest

luisleo526 and others added 3 commits October 28, 2026 06:54

tests: frozen trade-history reference regenerated at the pinned base

bab2b5f

The twelve artifacts are byte-identical to the previous freeze; only the base pin and the manifest line for it changed. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Merge main (482b000) into the trade-retirement branch

f7781cd

github-code-quality Bot found potential problems Oct 27, 2026

View reviewed changes

Comment thread tests/test_report_sink_trade_comparison.py

import json

from pathlib import Path

import re

import unittest

luisleo526 mentioned this pull request Oct 27, 2026

report sink: retire sealed closed-trade rows on sink routes when the module opts in #591

Closed

luisleo526 closed this Oct 27, 2026

Navigation Menu

Repository navigation

report sink: retire sealed closed-trade rows on sink routes when the module opts in - #583

report sink: retire sealed closed-trade rows on sink routes when the module opts in#583
luisleo526 wants to merge 11 commits into
mainfrom
report-sink/trade-retirement

luisleo526 commented Oct 27, 2026 •

edited

Loading

Uh oh!

luisleo526 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

luisleo526 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

What does NOT change

Tests

Stated limits

How to verify

Uh oh!

luisleo526 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

luisleo526 commented Oct 27, 2026 •

edited

Loading