Skip to content

Commit 68c1a58

Browse files
committed
Add indexed ledger delivery and delta-v2 reports
Use indexed pending-delivery traversal, expose ledger storage status, and add the delta-v2 JSONL report format. Keep main's native risk declaration and update the native_execution_consumer.cpp and main.cpp line references in the CT7 row of docs/design/native-feature-parity.md. Register the existing report_jsonl_delta_v2_contract behavior under the required native_live_ledger_behavior_e2e CTest name on Linux x86-64; the other three ledger behaviors retain their names, so each population runs once.
1 parent 9e42fe9 commit 68c1a58

19 files changed

Lines changed: 1216 additions & 32 deletions

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,21 @@ version number follows semantic versioning over the surfaces the
132132
`copy`, `snapshot` or `restore`) instead of at creation, with the same typed failure
133133
`report_sink_failed` and `reason: arena_exhausted`. Scripts that create maps
134134
they never fill can now run where they used to fail at `map.new`.
135+
- The native runner seeks the next unsent delivery through the existing
136+
partial index, excluding delivered, journal-only and completed-failure
137+
history from the scheduler's cursor walk.
138+
- Native runner status reports database + WAL bytes, measured bytes per
139+
committed input, available disk and a growth-based exhaustion projection.
140+
`--ledger-warn-free-fraction F` (default 0.8) reports an advisory `WARNING`
141+
without changing readiness or exit status. The default allowance stays
142+
unlimited.
143+
- `--report-jsonl=delta-v2` adds versioned change-only report records with
144+
`ratios.as_of_input` on every record, outside the cumulative document, and
145+
a raw `--delta-base` bootstrap. The envelope bounds emitted bytes and rows
146+
serialized per input on the 10,000/100,000-input work populations.
147+
Ratio scheduling is a separate later change; whole-curve passes remain.
148+
Plain/full and v1 delta bytes are preserved. Ledger tables and checkpoint
149+
layout are unchanged.
135150

136151
## 1.9.1 — 2026-10-31
137152

‎docs/design/native-feature-parity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ Columns: **Feature** (adapter mechanism, cited) · **Native today** (`yes` / `pa
121121
| CT4 | Partial-bar view / open-bar lookahead guard | **no** — the open hook receives the *complete* script bar: `invoke_bar_open_callback(engine, bar, point)` native_execution_consumer.cpp:8552 with `engine.current_bar_ = open_view` native_execution_consumer.cpp:8603 (the complete bar unless `OpenOnly`) **Closed:** `current_partial_bar()` native_host.hpp:1093 is the lookahead-free bar so far, through the last path point consumed (a fill inside a segment reads it at the segment's origin), and `NativeOpenBarView::OpenOnly` native_run_spec.hpp:109 masks `on_native_bar_open`'s bar to `H = L = C = open` as presentation only — the complete bar is restored before the open match (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E3 | (single-source) The adapter needs the full bar, so the guard is opt-in. |
122122
| CT5 | Raw input observation before aggregation | **yes** — `on_native_input` native_host.hpp:895, `NativeInputContext` native_host.hpp:793-798 | K | — | O:F3 | (single-source row) |
123123
| CT6 | Calculate on order fills (scheduler `PineScheduler::recalculate` pine_scheduler_native.cpp:767, `coof_recalculation_due` pine_scheduler_native.cpp:733, the first-open chain `kFirstOpenLoopGuard` pine_scheduler_native.cpp:888; `begin_coof_recalc` pine_adapter.cpp:7148, `flush_coof_tail` pine_adapter.cpp:7570, `end_coof_recalc` pine_adapter.cpp:7205) | **partial** — `on_native_applied` native_host.hpp:958 fires mid-path after each fill native_execution_consumer.cpp:8452, native_execution_consumer.cpp:8452; commands are legal there `commands_allowed` native_execution_consumer.cpp:1980; a request born there is eligible on the unconsumed rest of the bar `born_on_remaining_path` native_execution_consumer.cpp:6908, `remaining_path_delivery` native_order.cpp:4224-4232. Missing: calculation re-entry, a documented chronology, a bar-so-far view **Closed:** `NativeCalculationTrigger::BarCloseAndFills` native_run_spec.hpp:96 drives one `OrderFill` recalculation per applied execution at that event's own cursor, delivered as `on_native_recalculate(..., OrderFill, cause)` and bounded by `max_recalculations_per_point` (`tests/test_native_calc_timing.cpp`). | K+A | Calculation timing | F:E4 O:F5 S:C2 | **Disagree:** F partial / O, S no; F, S v1 / O deferrable. R5-5: v1, sub-bar hook + chronology contract, adapter path unchanged. TV parts → CT11. |
124-
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8691; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | Calculation timing | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:78-92, the runner rejects it (`calc_on_every_tick` main.cpp:328), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:609. |
124+
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8691; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | Calculation timing | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:78-92, the runner rejects it (`calc_on_every_tick` main.cpp:336), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:609. |
125125
| CT8 | Historical intrabar calculation (per lower-TF sub-bar / magnifier sample) | **no** — with an `IntrabarPath` the callback still fires once per script-bar calculation (`invoke_callback` native_execution_consumer.cpp:9457). **Closed:** `EveryModeledPoint` recalculates at each intrabar sample, and `on_native_sub_bar` native_host.hpp:948 fires once after each retained lower-timeframe sub-bar's whole matching path (`tests/test_native_calc_timing.cpp`). | K | Calculation timing | F:E6 O:F8 S:C4 | — |
126126
| CT9 | Intrabar (magnifier) matching path + path-order policy (`volume_weighted_cap` pine_adapter.cpp:3125) | **yes** — `IntrabarPath` native_run_spec.hpp:378-434, delivery `deliver_confirmed_script` native_execution_consumer.cpp:8965-9085; `NativePathOrder` native_run_spec.hpp:335-339 | K+A | — | F:the typed delivery enums F:G5 O:the margin and cancellation comparison S:C4 | **Disagree:** S partial (it wants callback cadence → CT8). TV sampler choice is A: TradingView's own intrabars at its table's timeframe, each owned by the chart bar of its last minute, built from the finer feed (`tradingview_magnifier_bars` magnifier_intrabars.hpp:46, the TradingView intrabar import). |
127127
| CT10 | Batch → warmup → realtime forward execution | **yes** — `stream_begin` engine.hpp:2124, `stream_push_bar` engine.hpp:2130, `stream_end` engine.hpp:2134, `NativeRunPhase` native_host.hpp:41, C ABI `strategy_stream_begin` c_abi.cpp:840 | K+A | — | F:E8 O:J3 S:C5 | **Disagree:** S partial. R5-1: the lifecycle is native; Pine trade-start suppression and realtime-tail / probe rules (`trading_window_active` pine_strategy_commands.cpp:18-24, `prepare_native_begin` pine_strategy_host.cpp:554-722) are A. S's "neutral tail + trade-window policy" is single-source and unassigned (§4 Q8). |

‎docs/live-ledger-budget.md‎

Lines changed: 49 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ transaction's report state before its admission. The input is refused before
115115
growth if its cached pages plus that margin do not fit. A refused boundary
116116
input leaves database, WAL and SHM byte-identical, including repeated refusals.
117117
Charging the cumulative payload to the boundary input is intended
118-
(`Ledger::commit_input`, `runner/store.cpp:1099`). A boundary input can therefore
118+
(`Ledger::commit_input`, `runner/store.cpp:1140`). A boundary input can therefore
119119
hit a storage stop when the input alone would fit, up to the cumulative
120120
payload's reservation earlier than strictly necessary. This cost buys the
121121
guarantee that no input commits with a checkpoint admitted without room in its
@@ -142,8 +142,8 @@ Startup and recovery checkpoints also admit their actual cached pages before
142142
COMMIT, using the configured budget without an input reservation. They are
143143
optional on refusal and publish the same diagnostic. This is intended for both
144144
startup and post-replay checkpoints (`Ledger::checkpoint_recovered_report`,
145-
`runner/store.cpp:1225`; the post-replay call is `checkpoint_recovered_report`,
146-
`runner/main.cpp:1137`): with a large budget, either can spend most of the
145+
`runner/store.cpp:1266`; the post-replay call is `checkpoint_recovered_report`,
146+
`runner/main.cpp:1147`): with a large budget, either can spend most of the
147147
remaining headroom on an O(report) payload before the first live input. The run
148148
can stop sooner, never over budget. Opening files already
149149
over a newly supplied finite budget refuses with `LedgerBudgetExceeded` before
@@ -224,3 +224,49 @@ and prints request, actual growth, limit and admission result. Migration rows
224224
cover a ledger produced by the previous runner under a finite budget and a
225225
fixture with 4 MB of historical action payload under 8 MiB of upgrade headroom,
226226
with brute-force recounts and source-row comparisons.
227+
228+
## Advisory disk observations and indexed delivery
229+
230+
The default allowance remains unlimited. `--ledger-warn-free-fraction F`
231+
is an advisory operator parameter for `run` and read-only `status`, with
232+
finite `0 < F <= 1` and default 0.8. The sole source default is
233+
`kDefaultLedgerWarnFreeFraction` in `runner/ledger_observability.hpp`;
234+
neither this option nor its measurements are persisted.
235+
236+
Service output adds `metrics.ledger_storage`; `status` adds `ledger_storage`.
237+
`used_bytes` measures database plus WAL file sizes, excluding SHM.
238+
`bytes_per_input` divides that actual footprint by the committed input count;
239+
it includes fixed ledger overhead and is null at cursor zero.
240+
`free_disk_bytes` uses the filesystem's space available to this process.
241+
`growth_bytes_per_second` measures positive net footprint growth since recovery
242+
finished in this invocation, divided by elapsed steady-clock time.
243+
`seconds_to_disk_exhaustion` divides available free disk by that rate.
244+
`growth_start_bytes` and `growth_elapsed_seconds` expose the interval.
245+
Unknown measurements, absent positive growth and a one-shot status query have
246+
null rate/projection. This is a running observation, not a capacity promise:
247+
WAL reuse, checkpointing, future row sizes and input rates affect it.
248+
249+
`ledger_storage.state` becomes `WARNING` at
250+
`used_bytes >= F * free_disk_bytes`; otherwise it is `OK`, or `UNKNOWN` if
251+
measurement fails. The denominator is currently free disk, not total capacity.
252+
The warning publishes status immediately and never removes readiness, refuses
253+
an input, interrupts delivery or changes exit status. Existing explicit byte
254+
admission and its database + WAL + SHM definition remain as described above.
255+
There is no new cache, WAL, transaction or ledger limit.
256+
257+
The delivery cursor now seeks `unsent=1` ordinals through the existing partial
258+
`delivery_state_unsent` index, then joins the selected event and route.
259+
Acknowledged, journal-only and completed-failure history is excluded from the
260+
cursor walk. Active retries and explicit persisted redelivery retain their
261+
existing ownership. This changes a query only, with no index or table change.
262+
263+
`--report-jsonl=delta-v2` also changes behavior without changing the database
264+
schema. It stores versioned report payloads in the existing report rows and
265+
keeps ratio freshness outside the cumulative document; every v2 record carries
266+
`ratios.as_of_input`. Its change-only envelope bounds emitted bytes and
267+
serialized rows per input on the 10,000/100,000-input work populations.
268+
Plain/full output and v1 delta retain their bytes and ratio scheduling. See the
269+
[runner report contract](../runner/README.md#delta-v2-report-stream).
270+
The current exact default still performs whole-curve ratio work, including
271+
with v2. Ratio scheduling is a separate later change. This change does not
272+
claim flat memory or bounded cumulative export.

‎docs/pages/streaming.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,3 +244,25 @@ separate surfaces and are not implied by using this lifecycle.
244244
245245
See [Lifecycle](@ref lifecycle) for handle ownership and
246246
[FFI from Python](@ref ffi_python) for the complete POD mirrors.
247+
248+
## Native runner report channel
249+
250+
The optional native runner mirrors reports after input commit. Plain
251+
`--report-jsonl` and `--report-jsonl=full` emit full cumulative documents;
252+
`--report-jsonl=delta` retains the v1 delta stream. The explicit new
253+
`--report-jsonl=delta-v2` emits change-only fields and array replacement
254+
suffixes, plus `kind` and ratio freshness outside the reconstructed document.
255+
Every v2 record carries `ratios.as_of_input`. Every input record starts with
256+
`input_cursor`; no warmup/replay record is emitted. A stale terminal cursor
257+
can receive one ratio-only `final` record, followed by the operational summary.
258+
259+
Use `report --delta-base --ledger PATH --at-input C` to bootstrap a v2
260+
consumer with the raw cumulative document and its ratio metadata, then apply
261+
cursor C+1. Ordinary `report` refreshes exact ratios. The change-only envelope
262+
bounds emitted bytes and serialized rows per input on the 10,000/100,000-input
263+
work populations. Ratio scheduling is a separate later change; v2 retains
264+
the existing ratio interval policy and whole-curve passes. The envelope does
265+
not bound report memory. The complete wire contract and compatibility
266+
notes are in the [native runner guide](../../runner/README.md#delta-v2-report-stream).
267+
The runner also reports advisory disk usage and exhaustion projections;
268+
`--ledger-warn-free-fraction F` defaults to 0.8 and never removes readiness.

‎runner/CMakeLists.txt‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -710,3 +710,42 @@ if(PINEFORGE_BUILD_TESTS AND PINEFORGE_BUILD_SOURCE_LAYER)
710710
$<TARGET_FILE:test_native_live_report_cli> old-layout-report)
711711
add_dependencies(test_native_live_report_cli live_contract_old_layout)
712712
endif()
713+
714+
if(PINEFORGE_BUILD_TESTS AND CMAKE_SYSTEM_NAME STREQUAL "Linux"
715+
AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|amd64|AMD64)$")
716+
# The real CLI entry and libraries, with external link observers only.
717+
# This target can also be linked against the frozen runner implementation.
718+
add_executable(pineforge-live-observed main.cpp
719+
${PROJECT_SOURCE_DIR}/tests/native_live_ledger_work_observer.cpp)
720+
target_link_libraries(pineforge-live-observed PRIVATE pineforge_live_startup ${CMAKE_DL_LIBS})
721+
target_compile_options(pineforge-live-observed PRIVATE -Wall -Wextra -Wpedantic -ffp-contract=off)
722+
set(_pf_live_ratio_symbol
723+
"_ZN9pineforge7metrics20compute_equity_statsEPK17pf_equity_point_sldRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEddld")
724+
target_link_options(pineforge-live-observed PRIVATE
725+
"LINKER:--wrap=sqlite3_step" "LINKER:--wrap=sqlite3_finalize"
726+
"LINKER:--wrap=${_pf_live_ratio_symbol}")
727+
add_library(native_live_ledger_work_strategy MODULE
728+
${PROJECT_SOURCE_DIR}/tests/native_live_ledger_work_strategy.cpp)
729+
target_link_libraries(native_live_ledger_work_strategy PRIVATE PineForge::pineforge)
730+
target_compile_options(native_live_ledger_work_strategy PRIVATE -fvisibility=hidden -ffp-contract=off -UNDEBUG)
731+
set_target_properties(native_live_ledger_work_strategy PROPERTIES PREFIX "")
732+
add_dependencies(pineforge-live-observed native_live_ledger_fixture_writer native_live_ledger_work_strategy)
733+
foreach(_pf_ledger_row delivery_pending_tail_work ledger_observability_status
734+
report_default_delta_work)
735+
add_test(NAME ${_pf_ledger_row} COMMAND ${Python3_EXECUTABLE}
736+
${PROJECT_SOURCE_DIR}/tests/native_live_ledger_behavior_e2e.py ${_pf_ledger_row}
737+
--runner $<TARGET_FILE:pineforge-live-observed>
738+
--strategy $<TARGET_FILE:native_live_ledger_work_strategy>
739+
--fixture-writer $<TARGET_FILE:native_live_ledger_fixture_writer>
740+
--artifacts ${PROJECT_BINARY_DIR}/ledger-behavior-artifacts)
741+
set_tests_properties(${_pf_ledger_row} PROPERTIES LABELS "slow;live_ledger" TIMEOUT 7200)
742+
endforeach()
743+
add_test(NAME native_live_ledger_behavior_e2e COMMAND ${Python3_EXECUTABLE}
744+
${PROJECT_SOURCE_DIR}/tests/native_live_ledger_behavior_e2e.py report_jsonl_delta_v2_contract
745+
--runner $<TARGET_FILE:pineforge-live-observed>
746+
--strategy $<TARGET_FILE:native_live_ledger_work_strategy>
747+
--fixture-writer $<TARGET_FILE:native_live_ledger_fixture_writer>
748+
--artifacts ${PROJECT_BINARY_DIR}/ledger-behavior-artifacts)
749+
set_tests_properties(native_live_ledger_behavior_e2e PROPERTIES
750+
LABELS "slow;live_ledger" TIMEOUT 7200)
751+
endif()

0 commit comments

Comments
 (0)