Skip to content

Commit ff05c78

Browse files
authored
memory: state the report-memory bound exactly and add the proofs for it (resident, disk, jailed arena) (#556)
memory: state the report-memory bound exactly and add the proofs for it (resident, disk, jailed arena) The memory-management page said the default sink route keeps the equity curve out of memory, but not what still grows with the bar count, and that a jailed run without --report-sink-fd has no sink. It now states, with sources: the resident equity-report buffers are bounded on the default unjailed sink route; the curve scratch costs 24 B per script bar on disk (6 MiB at 262,144 and 48 MiB at 2,097,152 bars) with the E output and the JSON artifact; in a jailed run the scratch is the RAM-backed arena, so the same 24 B per script bar are memory there, sized by the hosting wrapper and failing closed with arena_exhausted; sink-off callers keep the native curve plus an owning report copy. "O(1) in bars" is not claimed for the jailed arena or the sink-off path. Removed text, named. The page text below was introduced by 6d2bf95 (docs: add the memory-management page, #531) and 30e9f9a (docs: memory-management page re-read against main after the 1.8.0 changes, #546): - the section "Use the report sink for bounded memory" is replaced by "Report memory: resident buffers, disk and the jailed arena" (its line citations were re-read and corrected); - the phrase "a bounded default route is planned, with no release date" and the sentence "Install a report sink; no release date for the default route" are removed from the report-memory table (the page documents the route as it is); - the two sink rows of that table are reworded to separate resident buffers from disk and arena growth; the sentences about the heap bound of the equity curve and the ledger, the sink file growth, and the legacy FULL sink route are rewritten into the new section, and the release container's default profile sentence of 30e9f9a is replaced by the route table. Tests: a two-size resident proof for the report scratch curve (N = 262,144 and 8N script bars, RSS growth within 1 MiB, scratch growth exactly 24 B per added bar, metric block bit-equal to the sink-off result); a corpus projection test that the temporary-sink route and the sink-off route give bit-identical metrics and diagnostics on real modules (it replaces the named-sink summary comparison on corpus modules that this test did before, and still skips when its manifest variable is unset). Both report-memory rows are pinned as slow rows (61 pinned names) and registered before the CR-150 include block. The binary-route peak-RSS row is left out of this change: its 9-minute wall exceeds the CI leg budget. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> ci: count the array-growth slow row with the report-memory slow row (62 pinned names) Both the report-memory change and the paged array store added one pinned slow row each on top of 60, and git merged the two identical edits of the count into 61. The pinned set now holds 62 names: 43 from the PINEFORGE_PR_SLOW_TESTS loop and 19 direct CMake LABELS assignments. The test assertion on len(PINNED_SLOW_TEST_ROWS) and the four sentences in docs/ci.md that state the count and its split (introduced by 3ea5a0d, ci: split the long test rows, shard the full CI legs and run them on the larger hosted runners, #544, and rewritten by the report-memory change) are updated to 62 and 43 + 19. No other text is removed. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> ci: update the slow-row count literals in test_ci_preflight to 62, 43 and 19 test_documented_slow_row_counts_match_literal_population pins the three numbers that docs/ci.md states for the pinned slow rows (all, from the PINEFORGE_PR_SLOW_TESTS loop, direct LABELS assignments) and two sentences that repeat them. The previous commit moved the documented count from 61 to 62 (43 + 19) but left this test at 61, 43 and 18, so test_ci_preflight_slice_6 failed on every profile. The test now states 62, 43 and 19. Only those literals change. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> memory: move three docker/run_json.py citations after the bulk equity copy The bulk path-sink equity copy added code above the report-sink helpers in docker/run_json.py, so the three line ranges the memory-management page cites for _ReportSink, _build_sink_report and _prepare_stream_report no longer contained their symbols (doc-anchors: SYMMISS). The ranges now read 6183-6220, 7102-7148 and 6983-7003. Citation numbers only: no sentence is added, removed or reworded. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Also rewritten: the 60-name slow-row counts in docs/ci.md introduced by 3ea5a0d (ci: split the long test rows ..., #544), now 62 = 43 + 19 (the report-memory row and #547's array-growth row); the Pine product host row of the report-memory table in docs/pages/memory-management.md introduced by 27cd77b (engine 1.8.0: cohort tables ..., #541), reworded to separate resident buffers from disk and arena growth.
1 parent 6b33c47 commit ff05c78

8 files changed

Lines changed: 910 additions & 47 deletions

File tree

‎docs/ci.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -265,8 +265,8 @@ registry-name or exclusion-name fields; the default exclusion still adds
265265
selected discovery and strict `ctest-exclusion` checks.
266266

267267
**Slow-label bound, 2026-10-24:** `PINNED_SLOW_TEST_ROWS` in
268-
`scripts/ci_verify.py` pins all 60 source-labelled names: 43 from the
269-
`PINEFORGE_PR_SLOW_TESTS` loop and 17 direct CMake `LABELS` assignments; the four report slices replace one direct name (+3), and the report and command-after covers remain outside the slow set.
268+
`scripts/ci_verify.py` pins all 62 source-labelled names: 43 from the
269+
`PINEFORGE_PR_SLOW_TESTS` loop and 19 direct CMake `LABELS` assignments; the four report slices replace one direct name (+3), and the report and command-after covers remain outside the slow set.
270270
Every name listed by `ctest -N -L slow` must belong to that set; conditional
271271
registration may omit names. An unpinned name fails `ctest-exclusion` on
272272
plain and 1/1 runs, or `ctest-shard-discovery` for multiple shards, with the
@@ -1422,9 +1422,9 @@ fewer of the 43 than Release does. `live-sanitizers`, `live-tsan` and the
14221422
parity subset keep their existing selections.
14231423

14241424
**The slow-label inventory also includes rows with direct labels:**
1425-
The timing list contains 43 names, and 17 additional names are pinned for
1425+
The timing list contains 43 names, and 19 additional names are pinned for
14261426
direct `LABELS` assignments in `tests/CMakeLists.txt` and
1427-
`tests/cr150.d/*.cmake`, making 60 distinct pinned slow names.
1427+
`tests/cr150.d/*.cmake`, making 62 distinct pinned slow names.
14281428
`PINNED_SLOW_TEST_ROWS` bounds the discovered exclusion by those names,
14291429
independent of the registered count; each profile may register a subset.
14301430

‎docs/pages/memory-management.md‎

Lines changed: 90 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,9 @@
44

55
This page tells a host how much memory a run can use, which part of it is
66
bounded, and which part grows with the bars, the trades or the open orders of
7-
the strategy it runs. Each figure is one of three kinds: a compile-time
8-
constant of the engine, a documented admission policy value, or a formula
9-
evaluated by hand from the pages it names. None is an RSS measurement unless
10-
the text says it was measured. A figure that was not measured says so in its
7+
the strategy it runs. Figures are engine constants, documented admission
8+
policy values, or formulas evaluated by hand from the sources named. None is
9+
an RSS measurement unless the text says it was measured. A figure that was not measured says so in its
1110
row.
1211

1312
## Where a run's memory lives {#memory_places}
@@ -22,20 +21,27 @@ A default run with no report sink needs a writable, disk-backed scratch
2221
directory even when it makes no trade. The spilled trade rows are `O(T)` on
2322
disk with a small resident window (see below).
2423

25-
## Use the report sink for bounded memory {#memory_sink}
26-
27-
The default route of the C API, with no report sink, keeps the equity curve in process
28-
memory: 24 bytes per script bar, so `O(N)` heap. **Install a report sink to
29-
keep it out of memory.** With a sink the engine keeps only the editable tail
30-
of the curve in memory (`BacktestEngine::store_equity_point` overwrites the held
31-
point when the scratch curve is set, defined at
32-
`src/native_execution_consumer.cpp:13611`). Beginning a sink empties the in-memory vector
33-
(`equity_curve_` at `src/engine_report.cpp:1208`) and installs a
34-
`ScratchCurveWriter` (`src/engine_report.cpp:1211`) that holds at most one
35-
64 KiB batch of 2,730 points (class `ScratchCurveWriter`,
36-
`src/report_scratch.hpp:63`). This holds on the native route and on the Pine
37-
host's legacy FULL sink route alike. The sink file or stream grows with the
38-
run; its writer window is bounded.
24+
## Report memory: resident buffers, disk and the jailed arena {#memory_sink}
25+
26+
Default C/C++ and Python callers without a report sink keep the native equity
27+
curve on the heap: **24 B per script bar plus vector spare capacity**
28+
(`pf_equity_point_t` include/pineforge/pineforge.h:326-330;
29+
`equity_curve_` include/pineforge/engine.hpp:719).
30+
Building the default FULL report then adds an owning **24 B per script bar**
31+
copy (`equity_curve` src/engine_report.cpp:481-484).
32+
33+
On the default sink route, the engine releases the resident vector and installs
34+
a scratch writer (`equity_curve_` src/engine_report.cpp:1205-1211), keeping an editable
35+
tail for reporting (`fill_report_sink_metrics_section`
36+
src/engine_report.cpp:1460-1467).
37+
The writer batches **2,730 points of 24 B**, within a **64 KiB** window
38+
(`kScratchWindowPoints` src/report_scratch.hpp:18-20;
39+
`ScratchCurveWriter::State` src/report_scratch.cpp:154-161).
40+
Metrics re-read scratch through bounded windows
41+
(`compute_report_sink_metrics` src/engine_report.cpp:1439-1450;
42+
`ScratchCurveWindow` src/report_scratch.cpp:270-310).
43+
This bounds resident equity-report buffers; the scratch backing and output
44+
still grow with the run.
3945

4046
One line to install it, on an idle handle before the run (it returns 0 when
4147
accepted):
@@ -59,10 +65,68 @@ command-line flags; it passes `--report-sink-fd` only when `PINEFORGE_REPORT_SIN
5965
is set), the same choice is `--report-sink PATH` (a file) or
6066
`--report-sink-fd N` (an inherited FIFO or stream). With a sink the returned
6167
report carries `NULL` trades and equity curve and the totals in its summary;
62-
read the records from the sink. The release container's default (unjailed)
63-
profile installs a temporary path sink, so its equity curve is not the `O(N)`
64-
heap case above; with `PINEFORGE_REPORT_SINK_FD` set it streams to that
65-
inherited descriptor instead. See [Report schema](@ref report_schema).
68+
read the records from the sink. See [Report schema](@ref report_schema).
69+
70+
The default **unjailed** `run_json` route, with a sink-capable module,
71+
installs a temporary path sink (`_new_report_sink`
72+
docker/run_json.py:5790-5822; `_ReportSink` docker/run_json.py:6183-6220).
73+
The full equity curve is **not resident**: with disk scratch and no arena
74+
attached, its resident equity-curve buffers are **O(1) in bars**.
75+
Binary curve scratch on **disk** still costs **24 B per script bar**
76+
(`flush` src/report_scratch.cpp:229-235):
77+
**6 MiB at 262,144 bars** and **48 MiB at 2,097,152 bars**, calculated payload
78+
sizes excluding other scratch and output.
79+
The sink's `E` output and the default route's JSON artifact also grow with
80+
the bar count: `_build_sink_report` docker/run_json.py:7102-7148 supplies lazy
81+
record arrays, and `_prepare_stream_report` docker/run_json.py:6983-7003
82+
serializes them to a temporary artifact.
83+
Lazy serialization bounds buffers, not output bytes.
84+
The test that checks the resident claim runs with a fixed minimal table
85+
admission (one entry per table); the default bounded-memory table manifest,
86+
which sizes the id tables from the bar count, is outside this claim.
87+
88+
Whenever a scratch arena is attached (a **jailed run with `--report-sink-fd`**
89+
always, and any other run that inherits one), curve scratch instead consumes
90+
the **RAM-backed arena**, so those same **24 B per script bar are memory**.
91+
The fd setter attaches the arena (`attach_from_environment`
92+
src/engine_report.cpp:1067), and `ScratchFile::open`
93+
include/pineforge/detail/scratch_file.hpp:127-134 selects it.
94+
Linux jailed admission requires sealed memfd backing
95+
(`strategy_require_report_scratch_arena_v1` src/c_abi.cpp:1431-1448).
96+
The hosting wrapper supplies `PF_SCRATCH_ARENA_BYTES`
97+
src/scratch_arena.cpp:385-386.
98+
The engine never resizes that backing (the sizing comment above `ScratchArena`
99+
include/pineforge/detail/scratch_arena.hpp:71-110).
100+
Arena allocation rounds to **1 MiB chunks**
101+
(`kChunkBytes` include/pineforge/detail/scratch_arena.hpp:112;
102+
`grow` src/scratch_arena.cpp:297-309); metadata and other stores are additional.
103+
Bounded mapping windows do not remove that arena memory cost.
104+
Exhaustion fails closed with typed `report_sink_failed`
105+
(`ReportScratchIoError` include/pineforge/detail/scratch_arena.hpp:16-21),
106+
reason `arena_exhausted` src/scratch_arena.cpp:234-238.
107+
There is no RAM fallback (the comment on `ScratchFile`
108+
include/pineforge/detail/scratch_file.hpp:30-36).
109+
110+
A **jailed run without `--report-sink-fd` has no sink at all**:
111+
`_new_report_sink` docker/run_json.py:5790-5807 returns `None`, so it retains
112+
the native curve and owning FULL report copy described above. It also builds
113+
the whole report text in memory before writing stdout, and that text is the
114+
largest per-bar term of this route: measured at 8,000,000 bars (kernel peak
115+
RSS slope, stdout byte-equal), about five bytes per stdout byte (one byte of
116+
text plus a four-byte-per-character buffer), for example 363 B per bar for a
117+
module with 63 B per bar of stdout and 767 B per bar for one with 144 B per
118+
bar. This is `O(output)`: it is the report the caller asked for.
119+
**"O(1) in bars" is not claimed for the jailed arena or the sink-off path.**
120+
These report bounds also exclude input mappings, strategy histories and the
121+
other stores listed on this page.
122+
123+
For the plain `.bin` input route the input file is mapped, 48 B per bar of file
124+
payload (`_map_bar_file` docker/run_json.py:2685-2708), and the touched pages
125+
count toward resident size as the run reads them. A read of the source sites
126+
in `load_bars_bin` (docker/run_json.py:2779-2811) finds no further per-bar copy of the chart, but
127+
this page publishes no measured figure for the route and does not give the
128+
reading as a bound; the sink and arena paragraphs above govern the report
129+
bytes.
66130
67131
## What you can set {#memory_controls}
68132
@@ -220,10 +284,10 @@ or bounds the row.
220284

221285
| What | Grows with | Where it lives | Fix |
222286
| --- | --- | --- | --- |
223-
| C or C++ host with no sink, and jailed Python without `--report-sink-fd` (the default route of the C API): equity curve | `N`: 24 B per script bar | Heap (a vector, so spare capacity is extra) | Install a report sink (above); a bounded default route is planned, with no release date |
224-
| Same route (no sink): closed trades | `T`: 168 B plus text per trade | Scratch, with 64 descriptor rows resident and a read cache of up to 65,536 entries (312 B each, allocated as trades accumulate); the FULL report built from them is `O(T)` heap | Install a report sink; no release date for the default route |
225-
| Any report sink installed (native or Pine host, history streamed or not) | The heap is bounded for the equity curve (one held point and one 64 KiB batch) and for the ledger (64 descriptor rows and a 128-entry read cache). The trailing block of rows that share the last exit time stays resident only when history is streamed (`K` lots closed at one time, `O(K)`). The sink file and the scratch curve grow with the run | Heap, scratch and the sink file or stream | Not applicable |
226-
| Pine product host (`PineStrategyHost`) | It cannot select `STREAMED` or `SUMMARY_ONLY`. With a sink installed and the default report mode (the legacy route) its report storage is bounded as in the row above: the curve through the scratch writer (`ScratchCurveWriter` at `src/engine_report.cpp:1211`), selected by the Pine host's report policy (`KernelRecordedAtHostMarks` at `src/source/pine_adapter.cpp:3039`) and the install condition (`report_policy` at `src/engine_report.cpp:1205`); and the ledger at 64 descriptor rows (`kResidentRows` at `include/pineforge/trade_ledger.hpp:38`) with the 128-entry sink read cache. Adapter state (placement rows, settled bracket families, script containers) is separate and has its own rows below | Heap bounded for the report; adapter state as the rows below | Not applicable |
287+
| C/C++ or Python host with no sink, including jailed Python without `--report-sink-fd` (the default route of the C API): equity curve | `N`: 24 B per script bar plus vector spare capacity, then another 24 B per bar for the owning FULL report copy when the default FULL report is built, plus the whole report text held until written (`O(output)`, about five bytes per stdout byte on the jailed route; see [Report memory](#memory_sink)); sources in [Report memory](#memory_sink) | Heap | Install a report sink (no bounded default route exists in this release); account for disk scratch or jailed arena memory as above |
288+
| Same route (no sink): closed trades | `T`: 168 B plus text per trade | Scratch, with 64 descriptor rows resident and a read cache of up to 65,536 entries (312 B each, allocated as trades accumulate); the FULL report built from them is `O(T)` heap | Install a report sink |
289+
| Default report sink route (native or Pine host, history streamed or not) | Equity-report buffers are bounded (an editable tail and one 64 KiB writer batch, plus bounded metric-read windows); the ledger has 64 descriptor rows and a 128-entry read cache. The trailing block of rows that share the last exit time stays resident only when history is streamed (`K` lots closed at one time, `O(K)`). Binary curve scratch grows at 24 B per script bar; `E` output and the default path route's JSON artifact also grow | Disk scratch without an arena; RAM-backed scratch in the jailed arena, plus resident buffers and the sink file or stream; sources in [Report memory](#memory_sink) | Not applicable |
290+
| Pine product host (`PineStrategyHost`) | It cannot select `STREAMED` or `SUMMARY_ONLY`. With a sink installed and the default report mode (the legacy route) its resident equity buffers and ledger cache are bounded as in the row above: the curve through the scratch writer (`ScratchCurveWriter` at `src/engine_report.cpp:1211`), selected by the Pine host's report policy (`KernelRecordedAtHostMarks` at `src/source/pine_adapter.cpp:3039`) and the install condition (`report_policy` at `src/engine_report.cpp:1205`); and the ledger at 64 descriptor rows (`kResidentRows` at `include/pineforge/trade_ledger.hpp:38`) with the 128-entry sink read cache. Adapter state (placement rows, settled bracket families, script containers) is separate and has its own rows below | Resident equity buffers and ledger cache bounded; curve scratch grows on disk or in the jailed RAM-backed arena; adapter state as the rows below | Not applicable |
227291
| A touched dormant leg that is re-issued | One placement row per re-issue, until the position cycle ends or the origin closes; an untouched re-issue with the same exit id replaces its row in place | Heap | Planned, with no release date |
228292
| Settled bracket families | One fixed record per archived family, which grows with `T` on a bracket strategy; not measured | Heap | Not scheduled |
229293
| Pending-book reads | `R`: `O(R^2)` for a size or a late indexed copy, `O(P R^2)` for a full indexed walk of `P` projected rows, up to `O(R^3)` when `P = R`; this is time, and it was already so in v1.6.3 | CPU | Planned, with no release date |

‎scripts/ci_verify.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -909,12 +909,12 @@ def permitted_exclusion_skips(pinned: frozenset[str], host_platform: str) -> fro
909909
if may_skip(host_platform))
910910

911911

912-
# Static slow-label policy: the 43-name CMake loop plus 17
912+
# Static slow-label policy: the 43-name CMake loop plus 18
913913
# direct LABELS assignments. Never derive this bound from the run's registry.
914914
# Conditional registration may omit pins; every discovered slow name must
915915
# belong to this set. A new slow row requires an explicit policy change.
916916
# Each verifier and command-after split replaces one slow name with eight (+7).
917-
# Four report slices replace one direct name (+3); 43 loop + 17 direct = 60, covers unlabelled.
917+
# Four report slices replace one direct name (+3); 43 loop + 18 direct = 61, covers unlabelled.
918918
PINNED_SLOW_TEST_ROWS = frozenset((
919919
# tests/CMakeLists.txt: the timing list, labelled by its slow loop.
920920
'test_adapter_live_state_equivalence',
@@ -955,6 +955,8 @@ def permitted_exclusion_skips(pinned: frozenset[str], host_platform: str) -> fro
955955
'test_native_settlement_carry',
956956
# tests/CMakeLists.txt:7354.
957957
'test_run_json_selected_sink_memory_l9s',
958+
# tests/CMakeLists.txt:7354 (report-memory bound row).
959+
'test_report_sink_scratch_curve_resident',
958960
# tests/cr150.d/core-cr150-cr257.cmake:10.
959961
'test_cr257_source_exit_scratch_io_100k',
960962
# tests/cr150.d/core-cr150-l18.cmake:6,9.

‎scripts/test_ci_preflight.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2534,9 +2534,9 @@ def test_documented_slow_row_counts_match_literal_population(self):
25342534
inventory = re.findall(
25352535
r'pins all (\d+) source-labelled names: (\d+) from the\n'
25362536
r'`PINEFORGE_PR_SLOW_TESTS` loop and (\d+) direct CMake', text)
2537-
self.assertEqual(inventory, [('60', '43', '17')])
2538-
self.assertIn('The timing list contains 43 names, and 17 additional names', text)
2539-
self.assertIn('making 60 distinct pinned slow names.', text)
2537+
self.assertEqual(inventory, [('62', '43', '19')])
2538+
self.assertIn('The timing list contains 43 names, and 19 additional names', text)
2539+
self.assertIn('making 62 distinct pinned slow names.', text)
25402540

25412541
def test_floor_table_documentation_anchors_point_to_definition(self):
25422542
source = (ROOT / 'scripts/ci_verify.py').read_text().splitlines()

‎scripts/test_ci_verify.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4475,7 +4475,7 @@ def test_the_table_is_the_floor_and_profiles_outside_it_keep_their_own(self):
44754475
{name for name in ci_verify.PINNED_SLOW_TEST_ROWS
44764476
if name.startswith('test_native_command_after')},
44774477
{f'test_native_command_after_slice_{index}' for index in range(1, 9)})
4478-
self.assertEqual(len(ci_verify.PINNED_SLOW_TEST_ROWS), 61)
4478+
self.assertEqual(len(ci_verify.PINNED_SLOW_TEST_ROWS), 62)
44794479
self.assertEqual(LIVE_SANITIZERS_MIN_TESTS, 21)
44804480
for name, (full, pinned) in ci_verify.PRIVATE_FIXTURE_PROFILE_FLOORS.items():
44814481
with self.subTest(profile=name):

‎tests/CMakeLists.txt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7347,6 +7347,16 @@ if(PINEFORGE_BUILD_SOURCE_LAYER)
73477347
endif()
73487348
endif()
73497349

7350+
# Report-memory bound: fresh Linux processes at N and 8N, including a FULL-handle reuse.
7351+
if(PINEFORGE_BUILD_SOURCE_LAYER AND PINEFORGE_BUILD_TUTORIAL
7352+
AND CMAKE_SYSTEM_NAME STREQUAL "Linux" AND CMAKE_BUILD_TYPE STREQUAL "Release"
7353+
AND NOT PINEFORGE_ENABLE_SANITIZERS AND NOT PINEFORGE_ENABLE_COVERAGE)
7354+
add_test(NAME test_report_sink_scratch_curve_resident
7355+
COMMAND test_report_sink_scratch_curve --resident-bound)
7356+
set_tests_properties(test_report_sink_scratch_curve_resident PROPERTIES
7357+
LABELS "slow;mem" RUN_SERIAL TRUE)
7358+
endif()
7359+
73507360
# CR-150: one file per lane in tests/cr150.d/<lane>.cmake (see tests/cr150.d/README.md).
73517361
file(GLOB CR150_LANE_CMAKE CONFIGURE_DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/cr150.d/*.cmake")
73527362
list(SORT CR150_LANE_CMAKE)

0 commit comments

Comments
 (0)