Skip to content

Commit 7b59662

Browse files
luisleo526claude
andauthored
Confirmed-bar streams compute what the batch computes (#325)
* Add confirmed-bar stream equivalence scan * Align population handoffs to recorded chart boundaries * Preserve native stream refusals when grading evidence * Apply modeled bar policies to confirmed streams * Preserve observed input semantics through close callbacks * Recompare archived stream scan evidence without extraction * Avoid batch replay for refused stream configurations * Document backtest and live information limits * Refresh adapter gate pins for merged policies * Reanchor source documentation after integration * Pin modeled entry throttles and callback epochs * Retain confirmed input and online period closure facts * Refresh source documentation anchors * Preserve terminal cap assertions across observed input * Refresh documentation anchors for integrated inputs * Separate confirmed and observed cap continuations * Retain confirmed interval openings beyond the recent window * Pin confirmed and observed continuation contracts * Preserve already-flat report marks exactly * Apply sibling stop ordering to confirmed input Correct the look-ahead description: Opening sibling-stop arbitration can inspect a complete script candle’s high and low in history. Live aggregation has only the children received so far; the opening script-bar seed likewise cannot contain its later children. The measured sibling-stop policy uses the supplied confirmed bar; the unavailable future-child opening seed remains documented. * Refresh documentation anchors after integration * Add generated security stream equivalence regressions * Preserve flat-book report equity marks * Restore metals daily labels for observed streams * Clarify streaming fixes and bounded source history * Pin confirmed-stream order policy equivalence * Pin next-open slipped-short affordability * Refresh source documentation anchors * Correct terminal host citations * Point the stream known issue at its fix and record the change Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent e9f78a6 commit 7b59662

65 files changed

Lines changed: 10009 additions & 304 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,17 @@ version number follows semantic versioning over the surfaces the
2020
`applied_runtime.syminfo`, so a gridded run has its own fingerprint digest. The
2121
engine library is unchanged.
2222

23+
- **Confirmed-bar streams compute what the batch computes:** a native stream fed
24+
confirmed bars now applies the same TradingView-parity rules as a batch backtest over
25+
the same bars. Rules that were switched off in any stream are now chosen by the input's
26+
domain: historical semantics for batch and confirmed-bar input, observed-print semantics
27+
only for genuine tick input. Fixes the v1.0.0/v1.0.1 known issue where same-chart
28+
`request.security` stopped updating after the first realtime bar when the input and
29+
script timeframes were equal, and two stream-only fill differences (both-sided stop
30+
entries with default sizing; `process_orders_on_close` attribution). Batch results are
31+
unchanged. The streaming guide gains a "Backtest vs live" section listing what a
32+
backtest can see that a live run cannot.
33+
2334
- **Native runner routing and delivery:** webhooks are optional; `--webhook-routes`
2435
adds first-match per-action targets and payload `pineforge-native-order-action/v2`.
2536
New `actions`, `status` and offline `redeliver` commands expose the journal and

‎README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -223,8 +223,10 @@ feed format, recovery and execution limitations. The existing validation
223223
scoreboard below describes batch backtests; it does not certify new native
224224
live behavior or real broker fills.
225225
Known issue (v1.0.0, v1.0.1): a stream whose input and script timeframes are
226-
equal serves stale `request.security` values after the first realtime bar; see
227-
[Streaming known issues](docs/pages/streaming.md) for the workaround.
226+
equal serves stale `request.security` values after the first realtime bar.
227+
Fixed on main (#325); included in the next release. See
228+
[Streaming known issues](docs/pages/streaming.md#streaming_known_issues) for the
229+
workaround for those versions.
228230

229231
## Validation scoreboard
230232

‎docs/adr/0001-kernel-adapter-boundary.md‎

Lines changed: 33 additions & 33 deletions
Large diffs are not rendered by default.

‎docs/design/native-feature-parity.md‎

Lines changed: 79 additions & 79 deletions
Large diffs are not rendered by default.

‎docs/native-refactor-progress.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,7 @@ them for the record only. "Linux aarch64" is the maintainers' Cortex-A725 host.
173173

174174
| Item | What it is | Measured size | Deferred by | Prototype and evidence | Reopens when |
175175
| --- | --- | --- | --- | --- | --- |
176-
| D2-B, the quiet-bar protocol and one receipt read per bar | PERF-D2's lane: a per-bar bar-open request, a bulk fold of quiet driver points, the adapter's quiet-open predicate with lazy open bookkeeping, and a gate before the receipt poll. D2-A landed the hook declarations and the O(1) `has_command_after` (`src/native_execution_consumer.hpp:269`). PERF-L4 gated the read inside `observe_terminal_receipts` (`src/source/pine_adapter.cpp:5617`), which still runs about three times a bar | PERF-D2, Linux aarch64 (Cortex-A725), callgrind and GBench: about ×0.88 on the 14 probes (an estimate from a measured −13 to −14 % floor bound for the bar-open request alone); after PB the receipt poll still costs about 250 instructions a bar on the no-order floor. On this tree the quiet-bar probe asks the receipt gate 40,996 times and each once-per-bar-open gate 13,248 times: 3.09 reads a bar, 82 % of them stopped at the gate (macOS arm64, AppleClang 17, Release) | "R5 PERF-D2 DONE 2026-09-24 19:56" (D2-B after V19-D); "R5 SUPERVISOR RULING 2026-09-25 00:15 Taipei": "D2-B (quiet-bar protocol) waits for PERF-D3's verdict"; TRIAGE1 §D | An unpublished prototype branch and its patches. PERF-D3's S1 (next row) prototyped the request and the lazy opening | A post-release performance epoch. Done when the receipt gate is asked as often as the once-per-bar gates |
176+
| D2-B, the quiet-bar protocol and one receipt read per bar | PERF-D2's lane: a per-bar bar-open request, a bulk fold of quiet driver points, the adapter's quiet-open predicate with lazy open bookkeeping, and a gate before the receipt poll. D2-A landed the hook declarations and the O(1) `has_command_after` (`src/native_execution_consumer.hpp:269`). PERF-L4 gated the read inside `observe_terminal_receipts` (`src/source/pine_adapter.cpp:5628`), which still runs about three times a bar | PERF-D2, Linux aarch64 (Cortex-A725), callgrind and GBench: about ×0.88 on the 14 probes (an estimate from a measured −13 to −14 % floor bound for the bar-open request alone); after PB the receipt poll still costs about 250 instructions a bar on the no-order floor. On this tree the quiet-bar probe asks the receipt gate 40,996 times and each once-per-bar-open gate 13,248 times: 3.09 reads a bar, 82 % of them stopped at the gate (macOS arm64, AppleClang 17, Release) | "R5 PERF-D2 DONE 2026-09-24 19:56" (D2-B after V19-D); "R5 SUPERVISOR RULING 2026-09-25 00:15 Taipei": "D2-B (quiet-bar protocol) waits for PERF-D3's verdict"; TRIAGE1 §D | An unpublished prototype branch and its patches. PERF-D3's S1 (next row) prototyped the request and the lazy opening | A post-release performance epoch. Done when the receipt gate is asked as often as the once-per-bar gates |
177177
| S1a, kernel quiet-bar driving (PERF-D3) | A bar with nothing live, a flat book and a declined opening is driven in a quiet form: the kernel writes the bookkeeping its four price points would leave (ordinals, the last two driver marks, the floor, the point epochs) without walking them or calling the bar-open callback. Additive host cadence declarations; C hosts declare from their callback table; one C symbol under the PF_API checklist | PERF-D3, Linux aarch64 (Cortex-A725), GBench time and `perf stat` instructions, on the L3 tree `b33ea659`, S1a with S1b: no-order floor 0.764 / 0.660 in time (0.718 / 0.656 in instructions), 14 probes 0.958 / 0.921 (0.892 / 0.856), 100 slots 0.972 / 0.956 per slot. A bare kernel host that opts in: idle 0.545× (35 % fewer instructions a bar), market 0.81×, bracket 0.92×. Byte-identical: corpus 312/312, the 132-configuration battery with its hash columns | "FINDING PERF-D3 (filed for a post-release epoch per the owner's cap)" (ledger, 2026-09-24 16:58 UTC); "OWNER DECISION 2026-09-25 00:20 Taipei"; TRIAGE1 §D: "PERF-D3 S1/S2 (post-release epoch)" | An unpublished prototype branch (five commits on L3's tree) and its patches | The post-release epoch, with the owner calls PERF-D3 lists (the API under the existing ruling, the restated gate count) |
178178
| S1b, the Pine lazy opening | The Pine host records what its skipped opening and its uncalled input callback would have written, at the bar's first host callback. Member functions only, no layout change | Measured with S1a. A quiet bar then asks 15 gates, not 30, so the quiet-bar cost check `CHECK(asked >= 20 * o.boundary.size())` (`tests/test_adapter_quiet_bar.cpp:1054`) must be restated | As S1a | As S1a. The in-script hash witnesses caught the prototype's one miss (`awaiting_legacy_script_open_ms_`) | As S1a |
179179
| S1c, the held-position wake band | S1 for bars that hold a position with nothing live, 41 % of the median public slot's bars. The host declares a price band, and the kernel calls the opening only when the bar's range leaves it. The adapter derives the band from its margin model and declines when it cannot bound it | PERF-D3, a measurement-only upper bound without the band, which is not sound in general: 100 slots 0.776 / 0.716 per slot, 14 probes 0.830 / 0.761, cumulative against L3 with S1, S2 and D2-A | As S1a; the owner call "Fund the price band" | An unpublished measurement-only script | As S1a. Size L, parity risk high: TradingView's margin money rules |
@@ -183,7 +183,7 @@ them for the record only. "Linux aarch64" is the maintainers' Cortex-A725 host.
183183
| Compact journal records and a history ring buffer (L3) | Smaller command records and a ring buffer for the request core's history. `WorkingRequestCore::history` (`include/pineforge/native_order.hpp:1709`) hands out the whole vector and the core holds `history_` (`include/pineforge/native_order.hpp:2197`) inside `native_order_v7`, so both are a layout and API change | L3, Linux aarch64 (Cortex-A725), `perf` call graphs, inclusive shares: at most 3.68 % of bracket, 11.99 % of re-issue, 4.26 % of market and 3.42 % of the 14 probes (retiring the history alone: 1.64, 4.51, 2.53 and 1.00 %); only part is realisable, since records must still be written | "R5 L3 DONE + ACCEPTED 2026-09-24 22:10 Taipei", ruling 1: "DEFERRED to the PERF-D3 structural plan"; TRIAGE1 §D: "compact journal records" | Lane L3's code, merged by INT22 (#285); its profiles are unpublished | A `native_order` layout epoch |
184184
| A checked-values handoff for executions (L3) | `execution_values` runs twice, in `check_execution` (`src/native_order.cpp:4791`) and again in `apply_execution` (`src/native_order.cpp:4805`); handing the checked values across needs a public type, an API addition with no layout change | L3, as above: 2.21 % of bracket for the two together, about 1.1 % recoverable | L3 ruling 2: "a public checked-values handoff for executions (~1.1% bracket) -> DEFERRED" | As above | An owner decision on the API addition |
185185
| A chunked `trades_` store (L3) | Closed trades in chunks rather than one vector; `trades_` (`include/pineforge/engine.hpp:546`) is protected `BacktestEngine` state, so script ABI | L3, as above: on bracket, vector growth 0.64 %, `Trade` moves 0.74 % and `record_close_trade` 0.43 % (at most about 1.4 %); at most 0.35 % on the 14 probes | L3 ruling 4: "chunked trades_ store (ABI) -> DEFERRED" | As above | An `engine_script_run` epoch that changes the engine's protected layout |
186-
| A zoned chart-day memo (D2-D finding 4) | On a chart with a timezone, `libc_chart_day_key` (`src/source/pine_adapter.cpp:15372`) takes the process timezone lock through `ScopedTimezone` and calls `localtime_r` on every bar, because `on_bar_open` (`src/source/pine_adapter.cpp:18681`) reads `chart_day_key` (`src/source/pine_adapter.cpp:18902`) unconditionally. The non-UTC monthly Sharpe/Sortino walk, `month_key_local` (`src/engine_metrics.cpp:33`), pays one `localtime_r` per equity point. UTC charts use D2-D's memo, `civil_chart_day_key` (`src/source/pine_adapter.cpp:15356`) | Not measured on a zoned chart. The UTC path cost 134 instructions a bar before D2-D's memo and 68 after (D2-D, Linux aarch64, callgrind on the floor). PERF-K1: `ScopedTimezone` copies the zone name, which allocates for names of 16 or more characters under libstdc++ | TRIAGE1 §D: "D2-D finding 4 (the timezone `localtime_r` memo)", under the owner's cap. D2-D's report: an exact zoned memo "would need the zone's transition table, which belongs in `timezone.cpp`" | None; the UTC memo's witness is `tests/test_chart_day_memo.cpp` | A lane that owns `src/timezone.cpp` and builds an exact zone-transition memo (PERF-K1's Sitka case shows why a shortcut is not exact), byte-identical, with the zoned keys of `test_chart_day_memo` still equal to the computed ones |
186+
| A zoned chart-day memo (D2-D finding 4) | On a chart with a timezone, `libc_chart_day_key` (`src/source/pine_adapter.cpp:15382`) takes the process timezone lock through `ScopedTimezone` and calls `localtime_r` on every bar, because `on_bar_open` (`src/source/pine_adapter.cpp:18691`) reads `chart_day_key` (`src/source/pine_adapter.cpp:18912`) unconditionally. The non-UTC monthly Sharpe/Sortino walk, `month_key_local` (`src/engine_metrics.cpp:33`), pays one `localtime_r` per equity point. UTC charts use D2-D's memo, `civil_chart_day_key` (`src/source/pine_adapter.cpp:15366`) | Not measured on a zoned chart. The UTC path cost 134 instructions a bar before D2-D's memo and 68 after (D2-D, Linux aarch64, callgrind on the floor). PERF-K1: `ScopedTimezone` copies the zone name, which allocates for names of 16 or more characters under libstdc++ | TRIAGE1 §D: "D2-D finding 4 (the timezone `localtime_r` memo)", under the owner's cap. D2-D's report: an exact zoned memo "would need the zone's transition table, which belongs in `timezone.cpp`" | None; the UTC memo's witness is `tests/test_chart_day_memo.cpp` | A lane that owns `src/timezone.cpp` and builds an exact zone-transition memo (PERF-K1's Sitka case shows why a shortcut is not exact), byte-identical, with the zoned keys of `test_chart_day_memo` still equal to the computed ones |
187187
| Thread-local reads that remain per call | D2-C moved the three scopes' writes onto the pump's block, `pump_ambient` (`src/native_execution_consumer.hpp:444`). A reader still reaches the block through `tl_runtime_ambient` (`src/ta_extremes_volume.cpp:29`): `bar_context` (`src/ta_extremes_volume.cpp:82`), read by every `ExtremeRing::update` (`src/ta_extremes_volume.cpp:117`), which backs `ta.highest`, `ta.lowest`, `ta.highestbars` and `ta.lowestbars` and, through an embedded `Highest` and `Lowest`, `ta.stoch`, `ta.wpr` and `ta.range`; and `matching_day_partition` (`src/timeframe.cpp:605`), read by the default-anchored `ta.vwap` through `session_day_index`, by `crosses_boundary` on a DAY period (`timeframe.change("D")`) and by the session-period helpers. `decompose_ms_local` (`src/session_time.cpp:250`) reads two thread-locals per call on a zoned clock, and the prepared order storage's `free_blocks` (`src/native_order.cpp:605`) one per prepared mutation. `ema_na_warmup_flag` is read once per EMA, at its first compute | One thread-local access per reader call. AUDIT4, macOS arm64, a dlopen'd tutorial-MACD probe, counted by `_tlv_get_addr` breakpoints and by descriptor patching: 1.000 a bar for one `ta::Highest` and for one default `ta::VWAP`, 1.999 for one `pine_hour` on a New York clock, 0.000 on the no-order floor and with orders. On Linux a TLS-descriptor call costs about 20 instructions: D2-C's 13 → 4 calls a bar was 260 → 80 instructions (Linux aarch64, GCC 13, callgrind). INT23: 2.00 a bar on the zoned probe 064, 0.08 on probe 008 | None for the TA and day-partition readers: "RULING: INT23 ACCEPTED" (ledger, 2026-09-24 23:30 UTC) reads "dlopen TLS 13->0/bar", and INT23's report names only the timezone and order-core residuals. This row is their record | Unpublished (AUDIT4's scratch) | A byte-identical lane that hands these readers the pump's block without a thread-local read. `ta::bar_context()` is script ABI, so its public declaration stays as it is |
188188
| "audit-off mode", "LTO", "data-oriented layout" | Named in "OWNER DECISION 2026-09-25 00:20 Taipei" among the post-release structural items: "S1 bulk quiet bars, S2 per-strategy specialization, audit-off mode, LTO+PGO, data-oriented layout". No engine or campaign document defines, designs or measures the three | None | The owner decision | None | Struck from this inventory until a design note defines one of them |
189189

@@ -199,7 +199,7 @@ unless the retention is Full, and `presize_logs`
199199
(`src/native_execution_consumer.cpp:9460-9471`) sizes no journal under Window.
200200
`NativeRunSpec::event_retention` (`include/pineforge/native_run_spec.hpp:714`)
201201
defaults to Window, and the Pine adapter declares
202-
`NativeEventRetention::Window` (`src/source/pine_adapter.cpp:2494`). K4 still
202+
`NativeEventRetention::Window` (`src/source/pine_adapter.cpp:2505`). K4 still
203203
sizes Full runs (driver log and journal) and Commands runs (journal), and its
204204
witness, `tests/test_native_batch_log_presize.cpp`, runs under Full. PERF-K24's
205205
replay figure (14.9 % less CPU on Linux aarch64) describes the tree before V19-B. In
@@ -210,7 +210,7 @@ or more, and the same run under Full holds 66,000 points in one block of
210210
**PERF-L4 reads the receipts about three times a bar, not once.** PERF-L4's
211211
brief asked it to "read the receipts ONCE per bar instead of three times". The
212212
lane gated the read instead: `observe_terminal_receipts`
213-
(`src/source/pine_adapter.cpp:5617`) returns at its receipt gate when no
213+
(`src/source/pine_adapter.cpp:5628`) returns at its receipt gate when no
214214
command lies above its cursor. `every_gate_is_taken_both_ways`
215215
(`tests/test_adapter_quiet_bar.cpp:1061`) prints
216216
`gate ReceiptRead asked 40996 quiet 33552 live 7444`, against `asked 13248`
@@ -241,7 +241,7 @@ Mac's 489 and 412 and 1003 of 1249, and no Mac output moved.
241241

242242
- **PERF-L4's retired reader slots.** `event_high_water_reader_` and
243243
`terminal_receipt_high_water_reader_`
244-
(`include/pineforge/source/pine_adapter.hpp:2026-2027`), with their type
244+
(`include/pineforge/source/pine_adapter.hpp:2028-2029`), with their type
245245
`ReceiptHighWaterReader`, are storage nothing reads or writes since PERF-L4:
246246
they keep `PineExecutionAdapter`'s layout, which is part of
247247
`PineStrategyHost`'s and so of the generated script's ABI. They go at the next

‎docs/pages/streaming.md‎

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,48 @@ with the [ABI v4 live surface](@ref live_surface); the earlier Python runtime
2929
that did so is retired. Tick-stream fills at observed
3030
prices/times can differ from batch OHLC path assumptions.
3131

32+
## Backtest vs live {#backtest_vs_live}
33+
34+
Backtest and live are modes of one engine. Confirmed-bar streams use the same
35+
modeled OHLC and Pine order policies as backtests; only genuine observed trades
36+
use tick prices instead of the historical OHLC path. The following historical
37+
look-ahead information is unavailable to a continuing live run:
38+
39+
- A historical opening script-bar seed can contain the complete script candle's
40+
high and low. Live aggregation has only the children received so far and cannot
41+
include later children; sibling-stop ordering still applies to the confirmed
42+
policy bar already supplied, without projecting those future children.
43+
- Historical `request.security(..., lookahead_on)` can project a requested
44+
period's final OHLC before that period closes. Live publishes only information
45+
that has arrived, not the future period close.
46+
- Historical next-source-bar lookup can inspect a retained future candle.
47+
Live has no such candle; requested-context feeding also cannot inspect the
48+
next actual input timestamp before that input arrives.
49+
- A one-bar historical run can accept an undetected timeframe. Live startup
50+
needs explicit timeframe metadata or enough confirmed timestamps to infer
51+
cadence; both the adapter fallback and startup validation have this limit.
52+
- History knows the final retained source-bar index. A continuing warmup cannot
53+
know that eventual index, while realtime `barstate.islast` describes the latest
54+
available bar rather than hindsight's final bar.
55+
- Historical EOF can seal the final run/session boundary. Live needs an explicit
56+
end event or a known calendar boundary; provisional last-regular-session-bar
57+
facts cannot assume an unseen next bar or session closure.
58+
59+
These look-ahead differences are deliberate, like TradingView history versus
60+
realtime. They are not emulated by supplying future data to live execution.
61+
Batch also tolerates gappy historical exports, but a stream refuses a missing
62+
confirmed bar: the feed adapter must recover it from REST or stop, never invent
63+
a candle. `calc_on_order_fills` is refused in streams today because the native
64+
stream's close-only source callback contract does not support fill-triggered
65+
re-entry; this is future source-layer work, not an inherent look-ahead limit.
66+
67+
A current source-layer limitation remains for requested periods whose last
68+
expected child is absent for calendar reasons: batch can seal that period when
69+
the next tradable input opens, but a confirmed stream does not yet forward that
70+
deferred calendar boundary. Requested values can therefore publish late or
71+
remain pending. This is follow-up work, not a deliberate look-ahead difference;
72+
it does not relax the refusal of a missing in-session confirmed bar.
73+
3274
This is the runtime model used by a continuously running strategy:
3375

3476
1. Call #strategy_stream_begin with every confirmed historical input bar.
@@ -170,7 +212,9 @@ separate surfaces and are not implied by using this lifecycle.
170212
the first realtime bar, so its live results can differ from a backtest of
171213
the same bars. Batch backtests are not affected. Workaround: run the stream
172214
with a script timeframe larger than the input timeframe (for example input
173-
`1`, script `5`), or use the batch backtest. A fix is in progress.
215+
`1`, script `5`), or use the batch backtest. **Fixed on main (#325); included in
216+
the next release.** This change feeds each newly observed confirmed bar to the
217+
requested-series evaluator, including equal-timeframe and Heikin-Ashi requests.
174218
175219
See [Lifecycle](@ref lifecycle) for handle ownership and
176220
[FFI from Python](@ref ffi_python) for the complete POD mirrors.

0 commit comments

Comments
 (0)