Skip to content

Commit ae0835d

Browse files
committed
fix: commit native provider batches atomically and clarify stream integration
1 parent 8901e83 commit ae0835d

9 files changed

Lines changed: 174 additions & 64 deletions

File tree

‎docs/pages/index.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,10 +44,10 @@ consumption.
4444
- <b class="tab-title">I'm connecting a realtime feed</b>
4545
Start with **[Historical to realtime streaming](@ref streaming)** for
4646
the warmup, ordered-trade, clock, and report lifecycle, then run
47-
`tutorial/run_stream.py` against the bundled MACD strategy. Building the
48-
`pineforge-live` recompute runtime instead? Read
49-
**[ABI v4 live surface](@ref live_surface)** — streaming's lifecycle is
50-
not what it uses.
47+
`tutorial/run_stream.py` against the bundled MACD strategy. The optional
48+
native C++ runner is documented in `runner/README.md`. For the separate
49+
Python `pineforge-live` recompute project, read
50+
**[ABI v4 live surface](@ref live_surface)**.
5151

5252
- <b class="tab-title">I'm analysing backtest results</b>
5353
Read the **[Trading metrics reference](@ref metrics)** — every

‎docs/pages/live-surface.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
@tableofcontents
44

55
**ABI v4** (`PF_ABI_VERSION == 4`) appends 24 exports and two `pf_report_t`
6-
fields for `pineforge-live`, the recompute-based live runtime built on top
6+
fields for the separate Python `pineforge-live` recompute-based project built on top
77
of this engine (see [`pineforge-workflow-live/docs/superpowers/specs/2026-09-07-pineforge-live-design.md` §3](https://github.com/pineforge-4pass/pineforge-workflow-live)
88
for the full contract this surface serves). Every one of the 24 symbols is
99
**default off / read-only** and **never changes a historical run**: the
@@ -12,10 +12,11 @@ accessor is a pure read over state the engine already computed for its own
1212
internal use. Historical-identity is pinned by the L0 evidence in
1313
[Evidence](#live_surface_evidence).
1414

15-
**No new evaluator.** Nothing here adds a second broker model —
16-
`run_backtest_full` is still the only fill-deciding entry point. The live
17-
runtime works by *recomputing* it (see @ref streaming for why the
18-
`strategy_stream_*` lifecycle is not used instead).
15+
**No new evaluator in this surface.** The Python runtime recomputes
16+
`run_backtest_full`. The optional native C++ runner in this engine repository
17+
uses `strategy_stream_*` instead (see @ref streaming and `runner/README.md`).
18+
Its live extension API, journals and verification scope are separate from
19+
these recomputation controls.
1920

2021
## The 24 symbols
2122

‎docs/pages/streaming.md‎

Lines changed: 13 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -8,26 +8,25 @@ The handoff preserves broker state, position and equity, pending orders, Pine
88
series and variables, TA objects, `request.security()` evaluators, and a
99
partially formed higher-timeframe candle.
1010

11-
@note **`pineforge-live` does not use this lifecycle.** The live runtime's
12-
fill model is the batch broker graded by the parity campaign, not the
13-
streaming broker's fills-at-the-observed-print model (with its warmup gates
14-
that disable some batch rules). Instead of `strategy_stream_*`,
15-
`pineforge-live` recomputes `run_backtest_full` over the accumulated
16-
script-timeframe bar array on every settlement and every intra-bar probe,
17-
using the ABI v4 live-runtime flags (`strategy_set_realtime_tail`,
18-
`strategy_set_probe_suppress_tail_logic`, `strategy_set_path_order`,
19-
`strategy_set_broker_state_hash_recording`) to treat the array's last bar as
20-
a still-forming tail without changing `run_backtest_full`'s own semantics.
21-
See [ABI v4 live surface](@ref live_surface) for the full flag reference,
22-
default values, and evidence.
11+
The optional **native C++ `pineforge-live` executable** in this repository
12+
uses this lifecycle directly. It includes native transport, custom C++ parser
13+
plugins, SQLite recovery and order-action webhooks; see the
14+
[native runner guide](https://github.com/pineforge-4pass/pineforge-engine/blob/main/runner/README.md).
15+
16+
The separate [Python `pineforge-live` project](https://github.com/pineforge-4pass/pineforge-live)
17+
uses accumulated-bar `run_backtest_full` recomputation and the
18+
[ABI v4 live surface](@ref live_surface) instead. Its verification evidence
19+
and journals are separate from the native runner. Tick-stream fills at observed
20+
prices/times can differ from batch OHLC path assumptions.
2321

2422
This is the runtime model used by a continuously running strategy:
2523

2624
1. Call #strategy_stream_begin with every confirmed historical input bar.
2725
2. Feed normalized trades with #strategy_stream_push_tick, or pass one contiguous
2826
replay tape to #strategy_stream_push_ticks.
29-
3. Call #strategy_stream_advance_time at wall-clock boundaries, including
30-
quiet periods where no trade arrived.
27+
3. Call #strategy_stream_advance_time when the provider confirms no earlier
28+
ticks remain, including quiet periods. A wall clock alone does not prove
29+
that delayed source records have all arrived.
3130
4. Call #strategy_stream_end, then snapshot cumulative state with
3231
#strategy_stream_fill_report.
3332
5. Release report arrays with #report_free and the instance with

‎runner/CMakeLists.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@ if(PINEFORGE_BUILD_TESTS)
5252
find_package(Python3 REQUIRED COMPONENTS Interpreter)
5353
add_test(NAME native_live_e2e COMMAND ${Python3_EXECUTABLE}
5454
${PROJECT_SOURCE_DIR}/tests/native_live_e2e.py $<TARGET_FILE:pineforge-live>
55-
$<TARGET_FILE:native_live_example> $<TARGET_FILE:native_live_parser_example>)
55+
$<TARGET_FILE:native_live_example> $<TARGET_FILE:native_live_parser_example>
56+
$<TARGET_FILE:live_parser_fixture>)
5657
set_tests_properties(native_live_e2e PROPERTIES TIMEOUT 120)
5758
foreach(native_test test_native_live_store test_native_live_websocket)
5859
add_executable(${native_test} ${PROJECT_SOURCE_DIR}/tests/${native_test}.cpp)

‎runner/README.md‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,9 @@ instrument metadata. `--session` defaults to `24x7`, `--timezone` to `UTC`;
8181
`--chart-timezone` is independent and defaults to the engine UTC chart path.
8282
Repeat `--input TITLE=VALUE` and `--override KEY=VALUE` for strategy settings.
8383
Those setters belong to the compiled strategy; it must validate unsupported
84-
settings. The bundled example has no inputs or configurable overrides.
84+
settings. The bundled example has no inputs or configurable overrides. It also exports
85+
`run_backtest`/`run_backtest_full`/`report_free`, so the same C++ strategy can
86+
be loaded by either a batch or native live harness.
8587

8688
`--feed -` reads stdin. `--feed-url https://...` polls a **complete JSONL
8789
snapshot** (maximum 4 MiB) every `--poll-ms` (default 1000); `--check` fetches
@@ -142,7 +144,10 @@ Use `--parser your-parser.so --parser-config mapping.json` to translate a
142144
provider's complete message into normalized ticks, bars or time boundaries.
143145
The parser can emit zero events for a recognized control message, or several
144146
ordered events for a provider batch. The runner stages and validates the whole
145-
parser result before applying any events. An error emits no partial parse.
147+
parser result before applying any events. Parser errors emit no partial parse.
148+
All normalized events from one provider message then commit as one ledger
149+
input and outbox transaction; a failure during application discards the
150+
instance without committing any part of that message.
146151

147152
The public [C ABI header](../include/pineforge/live_parser.h) defines the
148153
versioned POD and callbacks. A C++ plugin exports:
@@ -173,7 +178,7 @@ order ledger or exposed in transport errors.
173178
## Durable orders and recovery
174179
175180
One process holds an exclusive lock beside the SQLite ledger. Each successful
176-
native input commits its normalized JSON, observable engine/stream hash and
181+
provider message commits its normalized event or batch, observable engine/stream hash and
177182
ordered immutable webhook events in one database transaction. Network delivery
178183
starts only after commit. A failed engine/database operation ends that
179184
process; it never continues with an advanced but uncommitted strategy.
@@ -187,12 +192,17 @@ and stream state, not arbitrary C++ private variables.
187192
188193
Files and HTTP snapshots normally repeat the full normalized event prefix;
189194
matching records are skipped, changed records are refused. `--from-input N`
190-
declares the zero-based first **normalized event** in a resumed file/stdin/WS
191-
tail; it cannot skip beyond the recorded count. Parser heartbeats do not
192-
consume an input index. HTTP snapshots always begin at index zero. The final
195+
declares the zero-based first **nonempty provider message** in a resumed
196+
file/stdin/WS tail; it cannot skip beyond the recorded count. A parser batch
197+
of multiple events uses one index, and heartbeats emitting no events use
198+
none. HTTP snapshots always begin at index zero. The final
193199
stdout JSON reports `inputs_committed`; use that cursor with a provider that
194-
can resume exactly. `--max-events N` counts newly committed normalized input
195-
events, not order actions, and is useful for bounded replay/restart checks.
200+
can resume exactly. `--max-events N` counts newly committed nonempty provider
201+
messages, not individual batch elements or order actions. It stops only
202+
between atomic messages. `last_tick_sequence` reports the last committed
203+
source tick sequence for provider-specific resumption. The built-in JSONL
204+
parser also accepts `{"type":"batch","events":[...]}` with 1..1024
205+
normalized tick/bar/time events.
196206
197207
The webhook payload schema is `pineforge-native-order-action/v1`:
198208
@@ -211,7 +221,7 @@ Delivery is **at least once**: a receiver may accept a request before the
211221
runner can persist its acknowledgment. Deduplicate `event_id` before trading.
212222
Attempts are saved before requests. The oldest unacknowledged event blocks
213223
later delivery; bounded exponential delays retry transient errors. Permanent
214-
HTTP refusal stops the process. After investigating, restart to retry, raising
224+
HTTP refusal, including redirects, stops the process. After investigating, restart to retry, raising
215225
`--max-attempts` beyond the durable count if its default of 8 was exhausted.
216226
This limit does not discard events or reset their IDs. There is no implicit
217227
skip/rewrite of a failed order.

‎runner/examples/strategy.cpp‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,20 @@ PF_API void strategy_set_input(void *s, const char *key, const char *value) {
2929
static_cast<NativeExample *>(s)->set_input(key, value);
3030
}
3131
PF_API void strategy_set_override(void *, const char *, const char *) {}
32+
PF_API void run_backtest(void *s, pineforge::Bar *bars, int n, pineforge::ReportC *out) {
33+
auto *strategy = static_cast<NativeExample *>(s);
34+
strategy->run(bars, n);
35+
strategy->fill_report(out);
36+
}
37+
PF_API void run_backtest_full(void *s, pineforge::Bar *bars, int n, const char *input_tf,
38+
const char *script_tf, int magnifier, int samples, int distribution,
39+
pineforge::ReportC *out) {
40+
auto *strategy = static_cast<NativeExample *>(s);
41+
strategy->run(bars, n, input_tf ? input_tf : "", script_tf ? script_tf : "", magnifier != 0,
42+
samples, static_cast<pineforge::MagnifierDistribution>(distribution));
43+
strategy->fill_report(out);
44+
}
45+
PF_API void report_free(pineforge::ReportC *out) { pineforge::BacktestEngine::free_report(out); }
3246
// Retain the core object that exports native streaming symbols from the static
3347
// engine library; no per-strategy streaming implementation is necessary.
3448
PF_API int native_example_abi_version() { return pf_abi_version(); }

‎runner/main.cpp‎

Lines changed: 59 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ constexpr std::size_t MAX_FRAME = 1024 * 1024;
3434
struct Config {
3535
std::string strategy, warmup, feed = "-", ledger, webhook, mode = "", input_tf = "1", script_tf,
3636
symbol, name = "strategy";
37-
std::string session = "24x7", timezone = "UTC", chart_timezone, secret_env, feed_url,
37+
std::string session = "24x7", timezone = "UTC", chart_timezone = "UTC", secret_env, feed_url,
3838
parser_path, parser_config_path, subscribe_path;
3939
std::vector<std::pair<std::string, std::string>> inputs, overrides, syminfo;
4040
std::uint64_t from_input = 0, max_events = 0, max_attempts = 8;
@@ -74,7 +74,8 @@ void validate_script_tf(const std::string &tf) {
7474
if (!tf.empty() && (tf.back() == 'D' || tf.back() == 'W')) {
7575
seconds = tf.back() == 'D' ? 86400 : 604800;
7676
digits.pop_back();
77-
if (digits.empty()) digits = "1";
77+
if (digits.empty())
78+
digits = "1";
7879
}
7980
auto count = unsigned_arg(digits);
8081
if (!count || count > static_cast<std::uint64_t>(INT_MAX) / seconds)
@@ -171,6 +172,8 @@ Config args(int argc, char **argv) {
171172
if (c.input_tf != "1")
172173
throw std::runtime_error("native runner input-tf currently must be 1 minute");
173174
validate_script_tf(c.script_tf);
175+
if (c.chart_timezone.empty())
176+
c.chart_timezone = "UTC";
174177
if (!c.feed_url.empty() && seen.count("--feed"))
175178
throw std::runtime_error("choose feed or feed-url");
176179
const bool websocket = c.feed_url.rfind("ws://", 0) == 0 || c.feed_url.rfind("wss://", 0) == 0;
@@ -500,6 +503,18 @@ std::vector<Event> actions(Strategy &s, const Config &c, const std::string &depl
500503
}
501504
return out;
502505
}
506+
void apply_record(Strategy &strategy, const Config &c, Cursor &cursor, const Json &record) {
507+
if (record.at("type").text() != "batch") {
508+
apply(strategy, c, cursor, record);
509+
return;
510+
}
511+
only_fields(record, {"type", "events"});
512+
const auto &events = record.at("events");
513+
if (events.kind != Json::Kind::Array || events.items.empty() || events.items.size() > 1024)
514+
throw std::runtime_error("normalized batch requires 1..1024 events");
515+
for (const auto &event : events.items)
516+
apply(strategy, c, cursor, event);
517+
}
503518
std::string identity(const Config &c, const std::string &warmup, const std::string &library,
504519
const std::string &parser_bytes, const std::string &parser_config) {
505520
Json j = Json::object({{"schema", Json::string("pineforge-native-ledger/v1")},
@@ -571,7 +586,7 @@ bool drain(Ledger &ledger, const HttpOptions &options, const Config &c, std::uin
571586
continue;
572587
}
573588
ledger.record_delivery_failure(e->id, result.error);
574-
if (result.status >= 400 && result.status < 500 && result.status != 408 &&
589+
if (result.status >= 300 && result.status < 500 && result.status != 408 &&
575590
result.status != 429)
576591
throw std::runtime_error("webhook receiver refused event; event remains queued");
577592
const auto delay =
@@ -604,11 +619,12 @@ int run(const Config &c) {
604619
throw std::runtime_error("strategy library changed during initialization");
605620
auto recorded = ledger.input_count();
606621
for (std::uint64_t i = 0; i < recorded; ++i) {
607-
if (stopped) return 130;
622+
if (stopped)
623+
return 130;
608624
auto row = ledger.input(i);
609625
if (!row)
610626
throw std::runtime_error("ledger input hole");
611-
apply(strategy, c, cursor, parse_json(row->canonical_json));
627+
apply_record(strategy, c, cursor, parse_json(row->canonical_json));
612628
auto events = actions(strategy, c, deployment);
613629
if (row->state_hash != std::to_string(strategy.hash(strategy.state)) ||
614630
events.size() != row->events.size())
@@ -632,27 +648,38 @@ int run(const Config &c) {
632648
std::uint64_t delivered = 0, processed = 0, replayed_prefix = 0;
633649
drain(ledger, webhook, c, delivered);
634650
auto consume_message = [&](const std::string &message, std::uint64_t &index) {
635-
for (const auto &frame : normalize(parser.get(), message)) {
636-
auto canonical = frame.dump();
637-
if (auto previous = ledger.input(index)) {
638-
if (previous->canonical_json != canonical)
639-
throw std::runtime_error("input conflicts with committed prefix");
640-
++replayed_prefix;
641-
} else {
642-
if (index != ledger.input_count())
643-
throw std::runtime_error("input sequence is not contiguous");
644-
apply(strategy, c, cursor, frame);
645-
auto events = actions(strategy, c, deployment);
646-
ledger.commit_input(index, canonical, strategy.hash(strategy.state), events);
647-
strategy.clear(strategy.state);
648-
++processed;
649-
if (!drain(ledger, webhook, c, delivered))
650-
return;
651-
}
652-
++index;
653-
if (stopped || (c.max_events && processed >= c.max_events))
651+
auto frames = normalize(parser.get(), message);
652+
if (frames.empty())
653+
return;
654+
Json frame;
655+
if (frames.size() == 1)
656+
frame = std::move(frames.front());
657+
else {
658+
Json events;
659+
events.kind = Json::Kind::Array;
660+
events.items = std::move(frames);
661+
frame = Json::object({{"type", Json::string("batch")}, {"events", std::move(events)}});
662+
}
663+
auto canonical = frame.dump();
664+
if (auto previous = ledger.input(index)) {
665+
if (previous->canonical_json != canonical)
666+
throw std::runtime_error("input conflicts with committed prefix");
667+
++replayed_prefix;
668+
} else {
669+
if (index != ledger.input_count())
670+
throw std::runtime_error("input sequence is not contiguous");
671+
// The entire provider message advances in memory before one
672+
// input/state/outbox transaction. Failure discards this instance;
673+
// interruption and delivery begin only after every event commits.
674+
apply_record(strategy, c, cursor, frame);
675+
auto events = actions(strategy, c, deployment);
676+
ledger.commit_input(index, canonical, strategy.hash(strategy.state), events);
677+
strategy.clear(strategy.state);
678+
++processed;
679+
if (!drain(ledger, webhook, c, delivered))
654680
return;
655681
}
682+
++index;
656683
};
657684
auto consume = [&](std::istream &in, std::uint64_t start, bool full_snapshot) {
658685
std::string row;
@@ -747,12 +774,14 @@ int run(const Config &c) {
747774
} while (!stopped);
748775
}
749776
auto pending = ledger.pending_count();
750-
std::cout << Json::object({{"deployment", Json::string(deployment)},
751-
{"inputs_committed", num(ledger.input_count())},
752-
{"inputs_processed", num(processed)},
753-
{"prefix_skipped", num(replayed_prefix)},
754-
{"webhooks_delivered", num(delivered)},
755-
{"webhooks_pending", num(pending)}})
777+
std::cout << Json::object(
778+
{{"deployment", Json::string(deployment)},
779+
{"inputs_committed", num(ledger.input_count())},
780+
{"inputs_processed", num(processed)},
781+
{"prefix_skipped", num(replayed_prefix)},
782+
{"webhooks_delivered", num(delivered)},
783+
{"webhooks_pending", num(pending)},
784+
{"last_tick_sequence", cursor.seen_tick ? num(cursor.tick_seq) : Json{}}})
756785
.dump()
757786
<< '\n';
758787
return stopped ? 130 : pending ? 2 : 0;

0 commit comments

Comments
 (0)