Skip to content

Commit 4342247

Browse files
luisleo526claude
andcommitted
live runner: a delta report stream by default, and report-path work that does not rescan closed trades
After every input the runner scanned the whole closed-trade prefix again and, with --report-jsonl, printed the entire cumulative report: quadratic CPU and output over a long run, plus a retained copy of the whole document. - ReportDeltas carries a closed-trade high-water mark, so extraction visits only the trades appended since the last update. The retained cumulative document exists only in full mode; the default keeps scalar state and array counts. - --report-jsonl now emits one pineforge-native-report-delta/v1 document per committed input, the same payload the ledger stores. The cumulative document is --report-jsonl=full. This changes the stream shape for consumers that read the cumulative document; in-repo consumers and the documentation are updated. - The native fill_report call still rebuilds the engine's trade and equity/metrics report for each input. Removing that term needs an engine-side carried accumulator and its own proof that batch reports stay byte for byte identical; the changelog says so. - tests/test_live_report.cpp compares the final cumulative report, applying the delta stream with the test's own applier, with the full-mode document; counts the closed trades visited over runs of N and 10N inputs and the bytes written in delta mode; and checks the option values. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 482b000 commit 4342247

9 files changed

Lines changed: 376 additions & 48 deletions

‎CHANGELOG.md‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,22 @@ diagnostics keys, described below.
7979
per-record sink parse and the dense-script close path are separate work, not in
8080
this release.
8181

82+
### Native live report stream
83+
84+
- **Delta JSONL is the default.** `pineforge-live run --report-jsonl` now
85+
emits one `pineforge-native-report-delta/v1` document for each committed
86+
input, matching the payload stored in the ledger. This deliberately changes
87+
the stream shape for existing consumers that read the old cumulative
88+
document; request `--report-jsonl=full` to retain that cumulative output.
89+
- **Per-input report-path work is flat in the appended rows.** The runner
90+
carries a closed-trade high-water mark and no longer scans the full closed
91+
trade prefix for every input. Delta output and the default runner state keep
92+
only scalar state and array counts, while full mode retains the cumulative
93+
document. The native `fill_report` call still rebuilds the engine's trade
94+
and equity/metrics report on each input; removing that remaining term needs
95+
an engine-side carried report accumulator and a separate batch-byte
96+
equivalence proof.
97+
8298
### Scratch placement (behaviour of 1.7.0 and later, restated)
8399

84100
- **Where scratch goes and when a run is refused.** The text is in
@@ -2336,8 +2352,10 @@ release either.
23362352
report. Each export re-verifies an append-only SHA-256 integrity chain over
23372353
the committed inputs, stored state hashes, actions and report rows, so a
23382354
ledger written before 1.2.0 needs one verified `run` replay to fill in its
2339-
reports and digests before it exports. `run --report-jsonl` mirrors each new
2340-
report to stdout; a closed pipe exits 1, not SIGPIPE, and loses no input.
2355+
reports and digests before it exports. The 1.2.0 runner mirrored each new
2356+
cumulative report to stdout; current 1.8.0 uses the delta stream by default
2357+
and `run --report-jsonl=full` retains that cumulative shape. A closed pipe
2358+
exits 1, not SIGPIPE, and loses no input.
23412359
The runner keeps tracing off and refuses to export a nonempty trace. A
23422360
confirmed-bar run's report is checked field by field against a batch
23432361
backtest of the same bars; the one tested difference is that a batch seals a

‎docs/design/native-feature-parity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ Columns: **Feature** (adapter mechanism, cited) · **Native today** (`yes` / `pa
121121
| CT4 | Partial-bar view / open-bar lookahead guard | **no** — the open hook receives the *complete* script bar: `invoke_bar_open_callback(engine, bar, point)` native_execution_consumer.cpp:8403 with `engine.current_bar_ = open_view` native_execution_consumer.cpp:8454 (the complete bar unless `OpenOnly`) **Closed:** `current_partial_bar()` native_host.hpp:1093 is the lookahead-free bar so far, through the last path point consumed (a fill inside a segment reads it at the segment's origin), and `NativeOpenBarView::OpenOnly` native_run_spec.hpp:109 masks `on_native_bar_open`'s bar to `H = L = C = open` as presentation only — the complete bar is restored before the open match (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E3 | (single-source) The adapter needs the full bar, so the guard is opt-in. |
122122
| CT5 | Raw input observation before aggregation | **yes** — `on_native_input` native_host.hpp:895, `NativeInputContext` native_host.hpp:793-798 | K | — | O:F3 | (single-source row) |
123123
| CT6 | Calculate on order fills (scheduler `PineScheduler::recalculate` pine_scheduler_native.cpp:767, `coof_recalculation_due` pine_scheduler_native.cpp:733, the first-open chain `kFirstOpenLoopGuard` pine_scheduler_native.cpp:888; `begin_coof_recalc` pine_adapter.cpp:7114, `flush_coof_tail` pine_adapter.cpp:7536, `end_coof_recalc` pine_adapter.cpp:7171) | **partial** — `on_native_applied` native_host.hpp:958 fires mid-path after each fill native_execution_consumer.cpp:8307, native_execution_consumer.cpp:8307; commands are legal there `commands_allowed` native_execution_consumer.cpp:1966; a request born there is eligible on the unconsumed rest of the bar `born_on_remaining_path` native_execution_consumer.cpp:6857, `remaining_path_delivery` native_order.cpp:4347-4355. Missing: calculation re-entry, a documented chronology, a bar-so-far view **Closed:** `NativeCalculationTrigger::BarCloseAndFills` native_run_spec.hpp:96 drives one `OrderFill` recalculation per applied execution at that event's own cursor, delivered as `on_native_recalculate(..., OrderFill, cause)` and bounded by `max_recalculations_per_point` (`tests/test_native_calc_timing.cpp`). | K+A | L5 | F:E4 O:F5 S:C2 | **Disagree:** F partial / O, S no; F, S v1 / O deferrable. R5-5: v1, sub-bar hook + chronology contract, adapter path unchanged. TV parts → CT11. |
124-
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8542; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:67-81, the runner rejects it (`calc_on_every_tick` main.cpp:259), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:608. |
124+
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8542; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:67-81, the runner rejects it (`calc_on_every_tick` main.cpp:272), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:608. |
125125
| CT8 | Historical intrabar calculation (per lower-TF sub-bar / magnifier sample) | **no** — with an `IntrabarPath` the callback still fires once per script-bar calculation (`invoke_callback` native_execution_consumer.cpp:9308). **Closed:** `EveryModeledPoint` recalculates at each intrabar sample, and `on_native_sub_bar` native_host.hpp:948 fires once after each retained lower-timeframe sub-bar's whole matching path (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E6 O:F8 S:C4 | — |
126126
| CT9 | Intrabar (magnifier) matching path + path-order policy (`volume_weighted_cap` pine_adapter.cpp:3120) | **yes** — `IntrabarPath` native_run_spec.hpp:362-403, delivery `deliver_confirmed_script` native_execution_consumer.cpp:8752-9018; `NativePathOrder` native_run_spec.hpp:335-339 | K+A | — | F:E7 F:G5 O:F7 S:C4 | **Disagree:** S partial (it wants callback cadence → CT8). TV sampler choice is A: TradingView's own intrabars at its table's timeframe, each owned by the chart bar of its last minute, built from the finer feed (`tradingview_magnifier_bars` magnifier_intrabars.hpp:46, R5 lane MAG-INTRABAR). |
127127
| CT10 | Batch → warmup → realtime forward execution | **yes** — `stream_begin` engine.hpp:1998, `stream_push_bar` engine.hpp:2004, `stream_end` engine.hpp:2008, `NativeRunPhase` native_host.hpp:41, C ABI `strategy_stream_begin` c_abi.cpp:781 | K+A | — | F:E8 O:J3 S:C5 | **Disagree:** S partial. R5-1: the lifecycle is native; Pine trade-start suppression and realtime-tail / probe rules (`trading_window_active` pine_strategy_commands.cpp:15-21, `prepare_native_begin` pine_strategy_host.cpp:548-702) are A. S's "neutral tail + trade-window policy" is single-source and unassigned (§4 Q8). |

‎docs/design/selected-window-wire-v1.3-revision-3.1.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ reason stays there; metrics, summary, trade and curve keys remain unchanged.
176176
The digest still has exactly 135 scalar leaves.
177177
The public writer's canonical-result validation and selected-window placement
178178
checks retain their existing scopes; `report_shape` is unchanged. The native
179-
runner's `native_report_json` (`runner/report.hpp:104`) exposes the separate
179+
runner's `native_report_json` (`runner/report.hpp:105`) exposes the separate
180180
native report format and has no existing diagnostics object designated
181181
outside this digest. It keeps undefined metrics as the string "NaN" in its
182182
own format, outside revision 3. The runner and C ABI carry no named reason;

‎runner/CMakeLists.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,11 @@ if(PINEFORGE_BUILD_TESTS)
7979
target_include_directories(test_live_json PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
8080
target_compile_options(test_live_json PRIVATE -UNDEBUG)
8181
add_test(NAME test_live_json COMMAND test_live_json)
82+
add_executable(test_live_report ${PROJECT_SOURCE_DIR}/tests/test_live_report.cpp)
83+
target_include_directories(test_live_report PRIVATE
84+
${CMAKE_CURRENT_SOURCE_DIR} ${PROJECT_SOURCE_DIR}/include ${PROJECT_BINARY_DIR}/include)
85+
target_compile_options(test_live_report PRIVATE -UNDEBUG)
86+
add_test(NAME test_live_report COMMAND test_live_report)
8287
add_executable(test_live_identity ${PROJECT_SOURCE_DIR}/tests/test_live_identity.cpp)
8388
target_link_libraries(test_live_identity PRIVATE pineforge_live_support)
8489
target_compile_options(test_live_identity PRIVATE -UNDEBUG)

‎runner/README.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,10 +44,12 @@ state hashes and immutable actions, even for an earlier requested cursor.
4444
A missing or mismatched digest refuses export. These SHA-256 digests detect
4545
ledger corruption, not edits by someone able to replace both data and digests;
4646
only `run` recomputes engine state by replay. `run --report-jsonl` mirrors
47-
each newly committed report to stdout, followed by the existing operational run
48-
summary; replayed source-prefix messages are not mirrored again. A closed
49-
stdout pipe fails clearly with exit 1, never SIGPIPE; the last input and report
50-
are already durable, and export remains available.
47+
each newly committed `pineforge-native-report-delta/v1` payload to stdout,
48+
followed by the existing operational run summary. Use
49+
`run --report-jsonl=full` when a cumulative `pineforge-native-report/v1`
50+
document is required. Replay of source-prefix messages is not mirrored again.
51+
A closed stdout pipe fails clearly with exit 1, never SIGPIPE; the last input
52+
and report are already durable, and export remains available.
5153

5254
The input, engine hash, actions, cumulative report and its integrity digest commit in one SQLite
5355
transaction. Append-only `pineforge-native-report-delta/v1` rows contain changed
@@ -57,7 +59,9 @@ canonical report JSON. Recovery verifies each input's engine hash, actions and
5759
exact report delta before delivery. Existing full-report rows remain immutable
5860
and readable; recovery verifies them against the reconstructed full report, then
5961
new inputs use deltas. This avoids quadratic report storage without changing the
60-
export schema or engine computation.
62+
export schema or engine computation. The default delta stream retains scalar
63+
state and array counts only; `--report-jsonl=full` retains the cumulative
64+
document needed for full JSONL output.
6165
Array suffix replacement assumes at most one provisional trailing element for
6266
equity points and broker hashes. The runner never enables tracing and refuses
6367
nonempty `trace`: traces can contain several records per bucket, so enabling

‎runner/main.cpp‎

Lines changed: 41 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
#include <dlfcn.h>
2424
#include <filesystem>
2525
#include <fstream>
26+
#include <functional>
2627
#include <iomanip>
2728
#include <iostream>
2829
#include <iterator>
@@ -41,6 +42,8 @@ static_assert(std::atomic<std::sig_atomic_t>::is_always_lock_free);
4142
void signal_stop(int) { stopped = 1; }
4243
constexpr std::size_t MAX_FRAME = 1024 * 1024;
4344

45+
enum class ReportJsonlMode { Off, Delta, Full };
46+
4447
struct Config {
4548
std::string strategy, warmup, feed = "-", ledger, webhook, mode = "", input_tf, script_tf,
4649
symbol, name = "strategy";
@@ -57,7 +60,8 @@ struct Config {
5760
std::uint64_t from_input = 0, max_events = 0, max_attempts = 8;
5861
std::uint64_t status_interval = 1, max_ledger_bytes = 0, feed_idle_timeout = 15, feed_message_timeout = 15;
5962
long poll_ms = 1000;
60-
bool check = false, allow_http = false, report_jsonl = false;
63+
bool check = false, allow_http = false;
64+
ReportJsonlMode report_jsonl = ReportJsonlMode::Off;
6165
};
6266

6367
void help() {
@@ -75,7 +79,8 @@ void help() {
7579
" --from-input N --max-events N --max-attempts 8\n"
7680
" --native-config FILE (strict native run specification)\n"
7781
" --check (one HTTP snapshot) --poll-ms 1000\n"
78-
" --report-jsonl (mirror committed cumulative reports to stdout)\n"
82+
" --report-jsonl (mirror committed report deltas to stdout)\n"
83+
" --report-jsonl=full (mirror cumulative reports to stdout)\n"
7984
" --status-file PATH --status-interval 1 (seconds, 1..300)\n"
8085
" --feed-idle-timeout 15 --feed-message-timeout 15 (seconds, 1..300)\n"
8186
" --control-dir PATH --max-ledger-bytes N (0 disables budget)\n"
@@ -131,13 +136,21 @@ Config args(int argc, char **argv) {
131136
std::string a = argv[i];
132137
if (a == "--parser" || a == "--parser-config")
133138
throw std::runtime_error("in-runner parsers were removed; use an external feed adapter (runner/README.md#feed-format)");
139+
if (a == "--report-jsonl" || a.rfind("--report-jsonl=", 0) == 0) {
140+
if (!seen.insert("--report-jsonl").second)
141+
throw std::runtime_error("duplicate option: --report-jsonl");
142+
c.explicit_flags.insert(a);
143+
if (a == "--report-jsonl")
144+
c.report_jsonl = ReportJsonlMode::Delta;
145+
else if (a == "--report-jsonl=full")
146+
c.report_jsonl = ReportJsonlMode::Full;
147+
else
148+
throw std::runtime_error("unknown option: " + a);
149+
continue;
150+
}
134151
if (a != "--input" && a != "--override" && a != "--syminfo" && !seen.insert(a).second)
135152
throw std::runtime_error("duplicate option: " + a);
136153
c.explicit_flags.insert(a);
137-
if (a == "--report-jsonl") {
138-
c.report_jsonl = true;
139-
continue;
140-
}
141154
if (a == "--check") {
142155
c.check = true;
143156
continue;
@@ -683,15 +696,21 @@ Json real(double value) {
683696
return Json::number(s.str());
684697
}
685698
std::string cumulative_report(Strategy& strategy, const std::string& deployment,
686-
std::uint64_t cursor, ReportDeltas& deltas) {
699+
std::uint64_t cursor, ReportDeltas& deltas,
700+
const std::function<void(const pf_report_t&, const std::string&,
701+
std::uint64_t)>& verify = {}) {
687702
pf_report_t report{};
688703
struct ReleaseReport {
689704
Strategy& strategy;
690705
pf_report_t& report;
691706
~ReleaseReport() { strategy.free_report(&report); }
692707
} release{strategy, report};
693708
strategy.check(strategy.fill_report(strategy.state, &report));
694-
return deltas.update(report, deployment, cursor, strategy.hash(strategy.state));
709+
const auto state_hash = strategy.hash(strategy.state);
710+
auto result = deltas.update(report, deployment, cursor, state_hash);
711+
if (verify)
712+
verify(report, result, state_hash);
713+
return result;
695714
}
696715
std::vector<Event> actions(Strategy &s, const Config &c, const std::string &deployment) {
697716
std::vector<Event> out;
@@ -913,8 +932,16 @@ int run(Config c) {
913932
Ledger ledger(c.ledger, deployment);
914933
ServiceFile service(c.status_file, c.status_interval * 1000);
915934
ledger.bind_routing(c.routing.stored_document());
916-
ReportDeltas report_deltas;
917-
ledger.verify_report(0, cumulative_report(strategy, deployment, 0, report_deltas), [&] { return report_deltas.json(); });
935+
ReportDeltas report_deltas(c.report_jsonl == ReportJsonlMode::Full);
936+
const auto verify_report = [&](std::uint64_t cursor) {
937+
cumulative_report(strategy, deployment, cursor, report_deltas,
938+
[&](const pf_report_t& native, const std::string& payload, std::uint64_t state_hash) {
939+
ledger.verify_report(cursor, payload, [&] {
940+
return report_deltas.full_json(native, deployment, cursor, state_hash);
941+
});
942+
});
943+
};
944+
verify_report(0);
918945
Cursor cursor;
919946
cursor.bar_timestamp = warmup.back().timestamp;
920947
auto recorded = ledger.input_count();
@@ -942,7 +969,7 @@ int run(Config c) {
942969
if (events[k].id != row->events[k].id || events[k].payload != row->events[k].payload ||
943970
events[k].target_id != row->events[k].target_id || events[k].delivery_id != row->events[k].delivery_id)
944971
throw std::runtime_error("native replay order-action mismatch");
945-
ledger.verify_report(i + 1, cumulative_report(strategy, deployment, i + 1, report_deltas), [&] { return report_deltas.json(); });
972+
verify_report(i + 1);
946973
strategy.clear(strategy.state);
947974
recovering.members["metrics"].members["report_cursor"] = num(ledger.report_cursor());
948975
service.publish(recovering);
@@ -1029,8 +1056,9 @@ int run(Config c) {
10291056
strategy.clear(strategy.state);
10301057
++processed;
10311058
source_timestamp = source_time(frame, source_time);
1032-
if (c.report_jsonl) {
1033-
std::cout << report_deltas.json() << '\n' << std::flush;
1059+
if (c.report_jsonl != ReportJsonlMode::Off) {
1060+
std::cout << (c.report_jsonl == ReportJsonlMode::Full ? report_deltas.json() : report)
1061+
<< '\n' << std::flush;
10341062
if (!std::cout) throw std::runtime_error("report-jsonl stdout write failed; committed input remains durable");
10351063
}
10361064
}

0 commit comments

Comments
 (0)