Skip to content

Commit 1df103f

Browse files
luisleo526claude
andcommitted
live runner: verified cumulative report checkpoints, so report export reads a bounded tail
Exporting the report of an earlier cursor rebuilt it from the whole stored history, and the runner had no checkpoint structure. - The ledger writes an immutable cumulative report checkpoint in the same transaction as the input commit at every configured boundary (default 1024 inputs, --checkpoint-interval K). Report export starts from the nearest checkpoint at or below the cursor and applies only the deltas after it, so the rows read are at most the interval plus the checkpoint row. - A checkpoint is verified against the report integrity chain before use; a payload or witness failure records a typed diagnostic and falls back to full reconstruction, which returns the same bytes. Ledgers written by the previous version open unchanged and get their first checkpoint at the next boundary. - The checkpoint holds report state only: the engine has no state restore entry point, so restart recovery still replays committed strategy inputs and validates their state hashes and actions. This is stated in the documentation. - tests/test_native_live_checkpoint.cpp compares exports with an independent full reconstruction for every cursor, bounds the rows read for a history ten times longer, refuses a corrupted checkpoint and falls back, opens an old ledger, and checks commit atomicity. tests/test_native_live_checkpoint_measurement.cpp is a diagnostic row that prints ledger bytes, WAL bytes, peak RSS, open descriptors and recovery and export times for growing input counts; it asserts only the structural bound. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 482b000 commit 1df103f

10 files changed

Lines changed: 871 additions & 63 deletions

‎CHANGELOG.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,22 @@ diagnostics keys, described below.
7979
per-record sink parse and the dense-script close path are separate work, not in
8080
this release.
8181

82+
### Native live report checkpoints
83+
84+
The live runner now writes an immutable cumulative report checkpoint in the
85+
same transaction as every configured checkpoint boundary. Report export starts
86+
from the nearest verified checkpoint and applies only the following deltas,
87+
with a structural tail bound of the checkpoint interval plus the checkpoint
88+
row. A payload or integrity witness failure records a typed diagnostic and
89+
falls back to full report reconstruction. The default interval is 1024 inputs
90+
and can be changed with `--checkpoint-interval K`.
91+
92+
The checkpoint contains report state only. The engine has no state restore
93+
entry point, so restart recovery still replays committed strategy inputs and
94+
validates their state hashes and actions. The diagnostic measurement row prints
95+
the lab's file, WAL, RSS, descriptor and timing values; this release does not
96+
claim those unmeasured values in its documentation.
97+
8298
### Scratch placement (behaviour of 1.7.0 and later, restated)
8399

84100
- **Where scratch goes and when a run is refused.** The text is in

‎docs/design/native-feature-parity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ Columns: **Feature** (adapter mechanism, cited) · **Native today** (`yes` / `pa
121121
| CT4 | Partial-bar view / open-bar lookahead guard | **no** — the open hook receives the *complete* script bar: `invoke_bar_open_callback(engine, bar, point)` native_execution_consumer.cpp:8403 with `engine.current_bar_ = open_view` native_execution_consumer.cpp:8454 (the complete bar unless `OpenOnly`) **Closed:** `current_partial_bar()` native_host.hpp:1093 is the lookahead-free bar so far, through the last path point consumed (a fill inside a segment reads it at the segment's origin), and `NativeOpenBarView::OpenOnly` native_run_spec.hpp:109 masks `on_native_bar_open`'s bar to `H = L = C = open` as presentation only — the complete bar is restored before the open match (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E3 | (single-source) The adapter needs the full bar, so the guard is opt-in. |
122122
| CT5 | Raw input observation before aggregation | **yes** — `on_native_input` native_host.hpp:895, `NativeInputContext` native_host.hpp:793-798 | K | — | O:F3 | (single-source row) |
123123
| CT6 | Calculate on order fills (scheduler `PineScheduler::recalculate` pine_scheduler_native.cpp:767, `coof_recalculation_due` pine_scheduler_native.cpp:733, the first-open chain `kFirstOpenLoopGuard` pine_scheduler_native.cpp:888; `begin_coof_recalc` pine_adapter.cpp:7114, `flush_coof_tail` pine_adapter.cpp:7536, `end_coof_recalc` pine_adapter.cpp:7171) | **partial** — `on_native_applied` native_host.hpp:958 fires mid-path after each fill native_execution_consumer.cpp:8307, native_execution_consumer.cpp:8307; commands are legal there `commands_allowed` native_execution_consumer.cpp:1966; a request born there is eligible on the unconsumed rest of the bar `born_on_remaining_path` native_execution_consumer.cpp:6857, `remaining_path_delivery` native_order.cpp:4347-4355. Missing: calculation re-entry, a documented chronology, a bar-so-far view **Closed:** `NativeCalculationTrigger::BarCloseAndFills` native_run_spec.hpp:96 drives one `OrderFill` recalculation per applied execution at that event's own cursor, delivered as `on_native_recalculate(..., OrderFill, cause)` and bounded by `max_recalculations_per_point` (`tests/test_native_calc_timing.cpp`). | K+A | L5 | F:E4 O:F5 S:C2 | **Disagree:** F partial / O, S no; F, S v1 / O deferrable. R5-5: v1, sub-bar hook + chronology contract, adapter path unchanged. TV parts → CT11. |
124-
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8542; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:67-81, the runner rejects it (`calc_on_every_tick` main.cpp:259), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:608. |
124+
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8542; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:67-81, the runner rejects it (`calc_on_every_tick` main.cpp:266), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:608. |
125125
| CT8 | Historical intrabar calculation (per lower-TF sub-bar / magnifier sample) | **no** — with an `IntrabarPath` the callback still fires once per script-bar calculation (`invoke_callback` native_execution_consumer.cpp:9308). **Closed:** `EveryModeledPoint` recalculates at each intrabar sample, and `on_native_sub_bar` native_host.hpp:948 fires once after each retained lower-timeframe sub-bar's whole matching path (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E6 O:F8 S:C4 | — |
126126
| CT9 | Intrabar (magnifier) matching path + path-order policy (`volume_weighted_cap` pine_adapter.cpp:3120) | **yes** — `IntrabarPath` native_run_spec.hpp:362-403, delivery `deliver_confirmed_script` native_execution_consumer.cpp:8752-9018; `NativePathOrder` native_run_spec.hpp:335-339 | K+A | — | F:E7 F:G5 O:F7 S:C4 | **Disagree:** S partial (it wants callback cadence → CT8). TV sampler choice is A: TradingView's own intrabars at its table's timeframe, each owned by the chart bar of its last minute, built from the finer feed (`tradingview_magnifier_bars` magnifier_intrabars.hpp:46, R5 lane MAG-INTRABAR). |
127127
| CT10 | Batch → warmup → realtime forward execution | **yes** — `stream_begin` engine.hpp:1998, `stream_push_bar` engine.hpp:2004, `stream_end` engine.hpp:2008, `NativeRunPhase` native_host.hpp:41, C ABI `strategy_stream_begin` c_abi.cpp:781 | K+A | — | F:E8 O:J3 S:C5 | **Disagree:** S partial. R5-1: the lifecycle is native; Pine trade-start suppression and realtime-tail / probe rules (`trading_window_active` pine_strategy_commands.cpp:15-21, `prepare_native_begin` pine_strategy_host.cpp:548-702) are A. S's "neutral tail + trade-window policy" is single-source and unassigned (§4 Q8). |

‎docs/pages/native-engine.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4745,6 +4745,15 @@ environment defaults for spec fields. Omitted CLI clock/symbol flags take the
47454745
file. Explicit `--input-tf`, `--script-tf`, `--timezone`, `--session`,
47464746
`--chart-timezone`, or `--symbol` must equal the file.
47474747

4748+
Live report export uses immutable cumulative checkpoints written with the
4749+
input transaction. The default checkpoint interval is 1024 committed inputs;
4750+
`--checkpoint-interval K` selects another positive interval. Export starts at
4751+
the nearest verified checkpoint and applies the following report deltas, so its
4752+
report-side tail is bounded by that interval and one checkpoint row. A failed
4753+
checkpoint witness falls back to full report reconstruction. Checkpoints do
4754+
not serialize strategy state: restart recovery still replays the committed
4755+
strategy inputs because the stream ABI has no state restore operation.
4756+
47484757
`input_mode` is the CLI `--mode` (`bars` or `ticks`), not a JSON key.
47494758
Timestamps in the spec, warmup CSV, and JSONL are Unix milliseconds.
47504759

‎runner/CMakeLists.txt‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,22 @@ if(PINEFORGE_BUILD_TESTS)
185185
add_test(NAME ${native_test} COMMAND ${native_test})
186186
endforeach()
187187
set_tests_properties(test_native_live_websocket PROPERTIES SKIP_RETURN_CODE 77)
188+
add_executable(test_native_live_checkpoint
189+
${PROJECT_SOURCE_DIR}/tests/test_native_live_checkpoint.cpp)
190+
target_link_libraries(test_native_live_checkpoint PRIVATE pineforge_live_legacy_test_support)
191+
target_compile_options(test_native_live_checkpoint PRIVATE -UNDEBUG)
192+
add_test(NAME test_native_live_checkpoint COMMAND test_native_live_checkpoint)
193+
if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
194+
add_executable(test_native_live_checkpoint_measurement
195+
${PROJECT_SOURCE_DIR}/tests/test_native_live_checkpoint_measurement.cpp)
196+
target_link_libraries(test_native_live_checkpoint_measurement PRIVATE
197+
pineforge_live_legacy_test_support)
198+
target_compile_options(test_native_live_checkpoint_measurement PRIVATE -UNDEBUG)
199+
add_test(NAME test_native_live_checkpoint_measurement
200+
COMMAND test_native_live_checkpoint_measurement)
201+
set_tests_properties(test_native_live_checkpoint_measurement PROPERTIES
202+
LABELS "slow;diagnostic" RUN_SERIAL TRUE)
203+
endif()
188204
if(PINEFORGE_BUILD_SOURCE_LAYER)
189205
add_test(NAME native_live_routing_e2e COMMAND ${Python3_EXECUTABLE}
190206
${PROJECT_SOURCE_DIR}/tests/native_live_routing_e2e.py $<TARGET_FILE:pineforge-live>

‎runner/README.md‎

Lines changed: 31 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -38,26 +38,33 @@ pineforge-live report --ledger orders.sqlite3 --deployment DEPLOYMENT --at-input
3838
is a committed-message cursor (the number of messages, not the zero-based input
3939
index, action sequence or bar count); 0 denotes warmup. Omit it for the latest
4040
committed report. An unavailable cursor or mismatched optional deployment exits
41-
1. Reading is safe while a runner owns the ledger. Export re-verifies the whole
42-
committed report-integrity chain in one read snapshot, including inputs, stored
43-
state hashes and immutable actions, even for an earlier requested cursor.
44-
A missing or mismatched digest refuses export. These SHA-256 digests detect
45-
ledger corruption, not edits by someone able to replace both data and digests;
46-
only `run` recomputes engine state by replay. `run --report-jsonl` mirrors
41+
1. Reading is safe while a runner owns the ledger. Export uses the nearest
42+
verified report checkpoint at or below the requested cursor and folds only the
43+
following deltas. The checkpoint carries the cumulative report bytes, its
44+
report-integrity chain value and a payload hash. A missing or mismatched
45+
checkpoint witness falls back to full report reconstruction and records a
46+
typed diagnostic. These SHA-256 digests detect ledger corruption, not edits by
47+
someone able to replace both data and digests; only `run` recomputes engine
48+
state by replay. `run --report-jsonl` mirrors
4749
each newly committed report to stdout, followed by the existing operational run
4850
summary; replayed source-prefix messages are not mirrored again. A closed
4951
stdout pipe fails clearly with exit 1, never SIGPIPE; the last input and report
5052
are already durable, and export remains available.
5153

5254
The input, engine hash, actions, cumulative report and its integrity digest commit in one SQLite
53-
transaction. Append-only `pineforge-native-report-delta/v1` rows contain changed
54-
scalar fields and appended/replaced array suffixes, not historical full-report
55-
copies. Export folds these rows through the selected cursor into byte-identical
56-
canonical report JSON. Recovery verifies each input's engine hash, actions and
57-
exact report delta before delivery. Existing full-report rows remain immutable
58-
and readable; recovery verifies them against the reconstructed full report, then
59-
new inputs use deltas. This avoids quadratic report storage without changing the
60-
export schema or engine computation.
55+
transaction. A report checkpoint is written in that transaction at every
56+
checkpoint interval. The default interval is 1024 committed inputs and can be
57+
changed with `--checkpoint-interval K`; smaller values trade more checkpoint
58+
rows and writes for a shorter report reconstruction tail. Append-only
59+
`pineforge-native-report-delta/v1` rows contain changed scalar fields and
60+
appended/replaced array suffixes, not historical full-report copies. Export
61+
folds these rows through the selected cursor into byte-identical canonical
62+
report JSON. Recovery still replays every committed strategy input because the
63+
strategy state has no restore API; it verifies each input's engine hash, actions
64+
before delivery, verifies the recomputed report at the
65+
checkpoint and later inputs, and uses the checkpoint only for the report-side
66+
prefix. This bounds report export work after a verified checkpoint, while
67+
strategy-state recovery remains replay work.
6168
Array suffix replacement assumes at most one provisional trailing element for
6269
equity points and broker hashes. The runner never enables tracing and refuses
6370
nonempty `trace`: traces can contain several records per bucket, so enabling
@@ -96,7 +103,8 @@ no report field is silently excluded from the confirmed-bar E2E comparison.
96103
pineforge-live run --strategy strategy.so --warmup history-15m.csv --script-tf 15 \
97104
--symbol EXCHANGE:SYMBOL --mode bars --feed - --ledger orders.sqlite3 \
98105
--status-file status.json --status-interval 1 --control-dir control \
99-
--feed-idle-timeout 15 --feed-message-timeout 15 --max-ledger-bytes 1073741824
106+
--feed-idle-timeout 15 --feed-message-timeout 15 --checkpoint-interval 1024 \
107+
--max-ledger-bytes 1073741824
100108
pineforge-live probe --status-file status.json --max-age 3
101109
pineforge-live probe --status-file status.json --max-age 3 --ready
102110
```
@@ -184,6 +192,9 @@ pineforge-live probe --status-file status.json --max-age 3 --ready
184192
and the bounded delivery drain can exceed it. Reserve free space for both;
185193
an actual I/O failure exits 1. No automatic pruning or report retention
186194
policy is applied.
195+
- `--checkpoint-interval K` writes a cumulative report checkpoint every K
196+
committed inputs. K must be positive and defaults to 1024. Checkpoints are
197+
report recovery aids; they do not serialize or restore strategy state.
187198

188199
### Live redelivery control files
189200

@@ -602,10 +613,12 @@ process; it never continues with an advanced but uncommitted strategy.
602613

603614
Restart uses the original strategy/warmup/settings, reconstructs the same
604615
native instance by replaying committed inputs, and compares hashes and every
605-
action payload before sending anything. Strategy private members are not
616+
action payload before sending anything. Report export can start at a verified
617+
checkpoint, but restart cannot restore strategy state from it and therefore
618+
still replays the committed strategy inputs. Strategy private members are not
606619
serialized: hand-written strategies must be deterministic and avoid external
607-
I/O/random state influencing decisions. The hash covers observable engine
608-
and stream state, not arbitrary C++ private variables.
620+
I/O/random state influencing decisions. The hash covers observable engine and
621+
stream state, not arbitrary C++ private variables.
609622

610623
Replay fails closed if changed timezone rules or session configuration make
611624
recorded input incompatible with the calendar. A cadence or input-gap refusal

‎runner/main.cpp‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ struct Config {
5555
std::optional<pineforge::native_calendar::SessionCalendar> input_calendar;
5656
mutable pineforge::native_calendar::SessionDayMemo input_calendar_memo;
5757
std::uint64_t from_input = 0, max_events = 0, max_attempts = 8;
58+
std::uint64_t checkpoint_interval = kDefaultReportCheckpointInterval;
5859
std::uint64_t status_interval = 1, max_ledger_bytes = 0, feed_idle_timeout = 15, feed_message_timeout = 15;
5960
long poll_ms = 1000;
6061
bool check = false, allow_http = false, report_jsonl = false;
@@ -73,6 +74,7 @@ void help() {
7374
" --webhook-secret-env NAME --allow-insecure-http\n"
7475
" --webhook-routes FILE (strict per-action routing; optional webhooks)\n"
7576
" --from-input N --max-events N --max-attempts 8\n"
77+
" --checkpoint-interval K (default 1024)\n"
7678
" --native-config FILE (strict native run specification)\n"
7779
" --check (one HTTP snapshot) --poll-ms 1000\n"
7880
" --report-jsonl (mirror committed cumulative reports to stdout)\n"
@@ -195,6 +197,11 @@ Config args(int argc, char **argv) {
195197
c.max_events = unsigned_arg(v);
196198
else if (a == "--max-attempts")
197199
c.max_attempts = unsigned_arg(v);
200+
else if (a == "--checkpoint-interval") {
201+
c.checkpoint_interval = unsigned_arg(v);
202+
if (!c.checkpoint_interval)
203+
throw std::runtime_error("checkpoint-interval must be positive");
204+
}
198205
else if (a == "--native-config")
199206
c.native_config = v;
200207
else if (a == "--status-file") c.status_file = v;
@@ -910,14 +917,22 @@ int run(Config c) {
910917
if (c.native.present)
911918
require_native_warmup(c.native, warmup);
912919
ControlDirectory controls(c.control_dir);
913-
Ledger ledger(c.ledger, deployment);
920+
LedgerOptions ledger_options;
921+
ledger_options.report_checkpoint_interval = c.checkpoint_interval;
922+
Ledger ledger(c.ledger, deployment, ledger_options);
914923
ServiceFile service(c.status_file, c.status_interval * 1000);
915924
ledger.bind_routing(c.routing.stored_document());
916925
ReportDeltas report_deltas;
917926
ledger.verify_report(0, cumulative_report(strategy, deployment, 0, report_deltas), [&] { return report_deltas.json(); });
918927
Cursor cursor;
919928
cursor.bar_timestamp = warmup.back().timestamp;
920929
auto recorded = ledger.input_count();
930+
std::optional<std::uint64_t> verified_report_checkpoint;
931+
try {
932+
verified_report_checkpoint = ledger.verified_report_checkpoint(recorded);
933+
} catch (const CheckpointFailure&) {
934+
verified_report_checkpoint.reset();
935+
}
921936
auto recovering = Json::object({{"deployment", Json::string(deployment)}, {"state", Json::string("recovering")},
922937
{"ready", Json::boolean(false)}, {"readiness", Json::object({
923938
{"validated_strategy_warmup", Json::boolean(true)}, {"recovered_ledger", Json::boolean(false)},
@@ -942,7 +957,13 @@ int run(Config c) {
942957
if (events[k].id != row->events[k].id || events[k].payload != row->events[k].payload ||
943958
events[k].target_id != row->events[k].target_id || events[k].delivery_id != row->events[k].delivery_id)
944959
throw std::runtime_error("native replay order-action mismatch");
945-
ledger.verify_report(i + 1, cumulative_report(strategy, deployment, i + 1, report_deltas), [&] { return report_deltas.json(); });
960+
auto report = cumulative_report(strategy, deployment, i + 1, report_deltas);
961+
if (!verified_report_checkpoint || i + 1 > *verified_report_checkpoint) {
962+
ledger.verify_report(i + 1, report, [&] { return report_deltas.json(); });
963+
} else if (i + 1 == *verified_report_checkpoint) {
964+
ledger.verify_report(i + 1, report, [&] { return report_deltas.json(); });
965+
ledger.verify_report_checkpoint(i + 1, report_deltas.json());
966+
}
946967
strategy.clear(strategy.state);
947968
recovering.members["metrics"].members["report_cursor"] = num(ledger.report_cursor());
948969
service.publish(recovering);

0 commit comments

Comments
 (0)