Skip to content

Commit af4c688

Browse files
committed
Document realtime streaming lifecycle
1 parent 9f69f96 commit af4c688

18 files changed

Lines changed: 488 additions & 56 deletions

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,13 +151,15 @@ int main(void) {
151151
}
152152
```
153153
154-
That's the entire integration. Every PineForge-compiled strategy `.so` exports the same 10 symbols — write your harness once, swap strategies forever.
154+
That's the entire batch integration. Every PineForge-compiled strategy `.so`
155+
exports the same stable ABI — write your harness once, swap strategies forever.
155156
156157
```bash
157158
cmake -B build -DCMAKE_BUILD_TYPE=Release
158159
cmake --build build -j
159160
ctest --test-dir build --output-on-failure # 39 tests, ~1 s
160161
bash tutorial/run.sh # MACD backtest end-to-end
162+
python3 tutorial/run_stream.py # OHLCV warmup → realtime trades
161163
```
162164

163165
## Documentation
@@ -167,6 +169,7 @@ bash tutorial/run.sh # MACD backtest end-to-end
167169
| 📖 **[cdocs.pineforge.dev](https://cdocs.pineforge.dev)** | Full C ABI reference, lifecycle, report schema, configuration knobs, magnifier, FFI bindings, ABI stability contract |
168170
| 🚀 **[Getting Started](https://cdocs.pineforge.dev/getting_started.html)** | 60-second build + install + smoke test |
169171
| 🧪 **[Tutorial: MACD on BTC/USDT](https://cdocs.pineforge.dev/tutorial_macd.html)** | End-to-end annotated walkthrough |
172+
| 📡 **[Historical → realtime streaming](https://cdocs.pineforge.dev/streaming.html)** | Preserve strategy and broker state while switching from OHLCV to ordered trades |
170173
| 📊 **[Trading metrics reference](https://cdocs.pineforge.dev/metrics.html)** | Every report metric (ABI v2): definitions, units, NaN rules, TV / quant-library validation status |
171174
| 🔌 **[FFI from Python](https://cdocs.pineforge.dev/ffi_python.html)** | Complete `ctypes` mirror — paste-ready |
172175
| 🦀 **[Calling from Rust](https://cdocs.pineforge.dev/examples_rust.html)** | Idiomatic `libloading` wrapper |

‎docs/Doxyfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ INPUT = ../include/pineforge/pineforge.h \
9090
pages/install.md \
9191
pages/integration-cmake.md \
9292
pages/lifecycle.md \
93+
pages/streaming.md \
9394
pages/report-schema.md \
9495
pages/metrics.md \
9596
pages/configuration.md \

‎docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ docs/
1717
│ ├── install.md
1818
│ ├── integration-cmake.md
1919
│ ├── lifecycle.md
20+
│ ├── streaming.md
2021
│ ├── report-schema.md
2122
│ ├── configuration.md
2223
│ ├── magnifier.md

‎docs/cheatsheet-runtime-and-execution.md‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -409,15 +409,14 @@ Audited gaps a forward/real-time executor must know (beyond per-order fills).
409409
are C++-accessor-only. Derive equity/profit-factor/Sharpe from the trades.
410410
- `pf_version_get()` / `pf_version_string()` for ABI gating.
411411

412-
## 3.10 Multi-run state (reused handle)
413-
414-
- **Reset per run:** `max_equity_`/`min_equity_`→`initial_capital_`, security
415-
feed counters, aggregator state.
416-
- **NOT reset:** `trades_`, `net_profit_sum_`, win/loss/even counts,
417-
`cons_loss_day_count_`, `risk_halted_`, `intraday_pnl_`, position/pyramid
418-
state, `pending_orders_`, `trail_best_price_`, `consumed_partial_exit_ids_`,
419-
trace buffer. Walk-forward on one handle **accumulates** — recreate the handle
420-
per independent run.
412+
## 3.10 Multi-run state and streams
413+
414+
- **One-shot `run_backtest*`:** resets broker, position, trades, equity,
415+
series/TA, risk, trace, security, and aggregator state before each run.
416+
Input/strategy/syminfo configuration remains attached to the handle.
417+
- **`strategy_stream_*`:** performs one historical warmup and deliberately
418+
preserves that exact state while ordered trades arrive. Use this lifecycle,
419+
not repeated one-shot calls, for continuous sessions.
421420

422421
## 3.11 Harness foot-gun
423422

‎docs/coverage.md‎

Lines changed: 23 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,16 @@
2424
> "no runtime module — Pine surface still supported" bucket below calls
2525
> it out.
2626
>
27-
> **Out of scope.** Visual / charting / alert APIs are not implemented
28-
> by this runtime regardless of consumer (PineForge is an offline
29-
> backtesting engine, not a renderer).
27+
> **Out of scope today.** Visual / charting / alert APIs are not implemented
28+
> by this runtime regardless of consumer. The runtime now accepts continuous
29+
> ordered-trade streams, but it does not yet emit alert events or render charts.
3030
3131
## Coverage summary
3232

3333

3434
| Category | Runtime status | What `libpineforge.a` owns |
3535
| --------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
36-
| Engine / strategy lifecycle | Supported | `BacktestEngine`, three `run(...)` overloads, bar loop, `on_bar(...)` hook, `ReportC` / `SecurityDiagC` reporting. |
36+
| Engine / strategy lifecycle | Supported | `BacktestEngine`, one-shot `run(...)` overloads, continuous historical-to-realtime streams, bar loop, raw-trade broker passes, `on_bar(...)` hook, and cumulative reporting. |
3737
| Strategy orders | Supported | `strategy_entry / order / exit / close / close_all / cancel / cancel_all` with OHLC-path fill resolution, OCA, pyramiding, slippage, commissions, margin gates, partial / FIFO-vs-ANY closes, trailing stops, and TV deferred-flip carry handling. |
3838
| Strategy state / accessors | Supported | Position state, equity / drawdown / runup tracking, win / loss counts, full closed- and open-trade accessor methods, intraday fill counter. |
3939
| Strategy risk | Supported | All six `strategy.risk.*` gates are wired (`engine_risk.cpp` + the fill/order gates): `allow_entry_in` direction allow-list, `max_position_size`, `max_drawdown` (abs / % of peak equity), `max_intraday_loss` (abs / % of equity), `max_cons_loss_days`, and `max_intraday_filled_orders` (latch-till-day-rollover cap-close). |
@@ -58,7 +58,7 @@
5858
## Public C ABI
5959

6060
`<pineforge/pineforge.h>` is the **single canonical consumer header**.
61-
Every compiled PineForge strategy `.so` exports exactly the 19 symbols
61+
Every compiled PineForge strategy `.so` exports exactly the 26 symbols
6262
declared there:
6363

6464

@@ -74,6 +74,12 @@ declared there:
7474
| `strategy_set_magnifier_volume_weighted` | Toggle volume-weighted magnifier |
7575
| `strategy_set_trace_enabled` | Toggle per-bar trace recording |
7676
| `strategy_set_trade_start_time` | Earliest Unix-ms at which order commands may fire |
77+
| `strategy_stream_begin` | Warm on confirmed OHLCV and enter realtime mode |
78+
| `strategy_stream_push_tick` | Push one ordered exchange trade |
79+
| `strategy_stream_push_ticks` | Push one contiguous ordered-trade array |
80+
| `strategy_stream_advance_time` | Confirm elapsed bars and materialize quiet intervals |
81+
| `strategy_stream_end` | End the realtime lifecycle |
82+
| `strategy_stream_fill_report` | Snapshot cumulative warmup + realtime state |
7783
| `strategy_set_chart_timezone` | Chart display TZ (intraday-day rollover gates) |
7884
| `strategy_set_syminfo_timezone` | Exchange TZ (`syminfo.timezone`) |
7985
| `strategy_set_syminfo_session` | Trading session string (`syminfo.session`) |
@@ -82,11 +88,13 @@ declared there:
8288
| `strategy_set_syminfo_metadata` | Inject fundamental / exchange metadata by Pine member name |
8389
| `strategy_get_last_error` | Error message from the most recent failed run |
8490
| `pf_version_get` | Runtime version (struct) |
91+
| `pf_abi_version` | Caller-allocated POD layout version |
8592
| `pf_version_string` | Runtime version (string) |
8693

8794

88-
POD types (`pf_bar_t`, `pf_trade_t`, `pf_report_t`, `pf_security_diag_t`,
89-
`pf_trace_entry_t`, `pf_version_t`) and the `pf_magnifier_distribution_t`
95+
POD types (`pf_bar_t`, `pf_trade_tick_t`, `pf_trade_t`, `pf_report_t`,
96+
the metrics/equity structs, `pf_security_diag_t`, `pf_trace_entry_t`,
97+
`pf_version_t`) and the `pf_magnifier_distribution_t`
9098
enum complete the surface. **Stability:** within the same
9199
`PINEFORGE_VERSION_MAJOR`, struct layouts and `extern "C"` signatures are
92100
append-only. New fields may be appended; existing fields are never
@@ -109,8 +117,8 @@ single `.hpp`):
109117

110118
| Module | Header | Source | Pine-facing role |
111119
| ------------------ | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
112-
| Public C ABI | `pineforge.h` | `c_abi.cpp` (+ layout `static_assert`s) | The 19 documented C symbols every compiled strategy `.so` exports. |
113-
| Engine | `engine.hpp` | `engine_run.cpp`, `engine_orders.cpp`, `engine_fills.cpp`, `engine_path_resolve.cpp`, `engine_strategy_commands.cpp`, `engine_trade_accessors.cpp`, `engine_security.cpp`, `engine_lower_tf.cpp`, `engine_risk.cpp`, `engine_report.cpp` | Strategy lifecycle, orders, fills, risk gating, reports, inputs / syminfo, magnifier loop, TF aggregation, `request.security` plumbing. |
120+
| Public C ABI | `pineforge.h` | `c_abi.cpp` (+ layout `static_assert`s) | The 26 documented C symbols every compiled strategy `.so` exports. |
121+
| Engine | `engine.hpp` | `engine_run.cpp`, `engine_stream.cpp`, `engine_orders.cpp`, `engine_fills.cpp`, `engine_path_resolve.cpp`, `engine_strategy_commands.cpp`, `engine_trade_accessors.cpp`, `engine_security.cpp`, `engine_lower_tf.cpp`, `engine_risk.cpp`, `engine_report.cpp` | One-shot and continuous lifecycle, orders, raw-trade/bar fills, risk, reports, inputs / syminfo, magnifier, TF aggregation, and `request.security` plumbing. |
114122
| Engine internals | `engine_internal.hpp` | (private cross-TU header) | `pineforge::internal::`* types and helpers shared between engine `.cpp` partitions; not part of the public ABI. |
115123
| Technical analysis | `ta.hpp` | `ta_moving_averages.cpp`, `ta_oscillators.cpp`, `ta_volatility_trend.cpp`, `ta_extremes_volume.cpp`, `ta_misc.cpp` | Official `ta.`* functions and series variables backed by stateful runtime classes with `compute` / `recompute`, plus `pivot_point_levels(...)` free function. |
116124
| Math | `math.hpp` | `math.cpp` | Inline `pine_random(...)` PRNG and rolling `math::Sum` class. |
@@ -767,20 +775,20 @@ carries a forward-looking assessment using these buckets:
767775
`fill`, `hline`, `bgcolor`, `barcolor`, `label.`*, `line.`*, `box.*`,
768776
`table.*`, `polyline.*`, `linefill.*`, `alert(...)`, `alertcondition(...)`.
769777
770-
- **Feasibility:** *Feasible* for plotting primitives (capture series + style metadata into the `ReportC` extension or a side-channel CSV / JSON for an external renderer); *Out of scope structurally* for live `alert(...)` because PineForge produces no realtime stream.
771-
- **Future story:** A "report-as-data" path is the obvious target — `plot(...)` and friends would write tagged time-series rows into a new diagnostics array on `ReportC`, and a Python harness would render them with Plotly / matplotlib. `alertcondition(...)` results could be returned as a list of `(bar_time, message)` triples. The graphics primitives (`label.new`, `line.new`, `box.new`) would need a small annotation runtime — straightforward but high surface area. Webhook / push-notification alerts stay out of scope.
778+
- **Feasibility:** *Feasible* for plotting primitives (capture series + style metadata into the `ReportC` extension or a side-channel CSV / JSON for an external renderer). Realtime `alert(...)` is now also feasible because the engine has an explicit realtime lifecycle, but it still needs alert frequency/dedup state, an event ABI, and delivery plumbing.
779+
- **Future story:** A "report-as-data" path is the obvious target — `plot(...)` and friends would write tagged time-series rows into a new diagnostics array on `ReportC`, and a Python harness would render them with Plotly / matplotlib. `alertcondition(...)` results could be returned as a list of `(bar_time, message)` triples. Alert delivery should drain deterministic engine events into an external JSON-only webhook adapter so network retries never mutate engine state.
772780
- **Why not done yet:** Backtests already produce `TradeC[]` and per-bar diagnostics; visual plotting has not been the unblocker for any user-facing strategy validation. It is a UX feature, not a correctness feature.
773781
774782
#### `varip` and realtime tick semantics
775783
776784
`varip` (persists across realtime ticks; resets on bar close in TV);
777785
`barstate.isrealtime`, `barstate.isnew` in realtime, `calc_on_every_tick`,
778-
`calc_on_order_fills`, live tick streams.
786+
`calc_on_order_fills`.
779787
780788
- **Feasibility:**
781789
- `varip` itself: *Feasible*. With the bar magnifier already simulating intrabar samples, `varip` could map onto sub-bar persistence (do not reset between magnifier ticks; reset only on bar close).
782-
- Realtime barstate flags + `calc_on_every_tick` + `calc_on_order_fills`: *Out of scope structurally* — they require a live data feed PineForge does not have.
783-
- **Future story:** `varip` mapping to magnifier sub-bar state is the right design; currently the support checker rejects `varip` outright but that is conservative rather than fundamental. Realtime semantics would require a streaming runtime that PineForge is not built for and does not aim to be.
790+
- Realtime barstate flags + `calc_on_every_tick` + `calc_on_order_fills`: *Feasible*. Ordered trades now reach the broker, but script recalculation still needs TradingView-style rollback, commit, and post-fill scheduling.
791+
- **Future story:** `varip` mapping to live per-tick persistence is the right design; currently the support checker rejects `varip` outright but that is conservative rather than fundamental. The new streaming lifecycle supplies the missing data/event boundary, while rollback and codegen support remain future work.
784792
- **Why not done yet:** `varip` use cases overlap heavily with what `var` already covers in batch mode. The realtime distinction TV makes is meaningful only when the data feed is live.
785793
786794
#### Import / export / library system
@@ -858,4 +866,4 @@ and [PineTS](https://github.com/LuxAlgo/PineTS), exercising the same
858866
surface across three independent engines (PineForge hits canonical
859867
*excellent* tier on 48 / 50 strategies vs PyneCore's 45 / 50; the 3
860868
PyneCore-only outliers all involve bracket / trail / partial exits, see
861-
`[benchmarks/results/summary.md](../benchmarks/results/summary.md)`).
869+
`[benchmarks/results/summary.md](../benchmarks/results/summary.md)`).

0 commit comments

Comments
 (0)