Skip to content

Confirmed-bar streams compute what the batch computes - #325

Merged
luisleo526 merged 30 commits into
mainfrom
lv/stream-gates
Oct 4, 2026
Merged

luisleo526 merged 30 commits into
mainfrom
lv/stream-gates

Conversation

@luisleo526

@luisleo526 luisleo526 commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Native confirmed-bar streams now compute the same strategy as the batch backtest over the same bars. "Backtest and live are two modes of one engine" holds for confirmed-bar input. Before this change, the Pine adapter switched off 46 TradingView-parity rules in any stream through stream_mode_ conditionals, so a stream could compute a different strategy from its own backtest. All changes are in the Pine adapter, scheduler and source host. The kernel is byte-identical: src/native*, include/pineforge/native* and src/engine* have no diff.

  • Input-domain rules instead of mode switches.
    • A single predicate, modeled_input(), gives historical (modeled) semantics to batch and to confirmed-bar stream input. Observed-print semantics are kept for genuine tick input only.
    • The mode/phase exclusions are removed, and every other qualification (price, size, book, order family, time, margin) is retained.
  • Online facts a stream needs:
    • a bounded, hashed provider of already-observed confirmed input bars;
    • online closure of requested periods;
    • a report cleanup that only clears a stale open profit on a genuinely flat book.
  • Stream defects fixed. These are pinned by generated batch-vs-stream equivalence tests that fail on main and pass here:
    • same-chart request.security stopped updating after the first realtime bar when the input and script timeframes were equal: one request went from 1308 batch vs 189 stream trades to 1308/1308, across all seven request shapes. This was a known issue of v1.0.0 and v1.0.1;
    • both-sided stop entries with default sizing: 276 vs 277 → 276/276;
    • process_orders_on_close entries were attributed one bar late: now identical.
  • Docs.
    • A "Backtest vs live" section lists what a backtest sees that a live run cannot yet see: genuine look-ahead, batch-only tolerance of gappy data, and calc_on_order_fills.
    • The stream known issue now reads "fixed on main (Confirmed-bar streams compute what the batch computes #325); next release".

Size and risk, for release planning

  • Behaviour conditionals on stream_mode_ in pine_adapter.cpp go from 49 on main to 1 predicate. git diff main...HEAD -- src include | grep -E '^\+.*stream_mode_' adds exactly two lines, and both form modeled_input() (pine_adapter.cpp:43-47). It is true in batch and for every confirmed-bar stream, and false only for genuine observed-print input, which has no batch counterpart: that is the documented tick-vs-OHLC difference, not a two-modes divergence. The other remaining uses are unchanged from main: rethrow on a stream failure (pine_strategy_host.cpp), hashing the flag (pine_state_hash.cpp), and setting it.
  • Source: 8 files, +199/−81. Tests: 50 files, +9,629, mostly generated equivalence fixtures. Docs: 6 files.
  • Batch identity: corpus 312/312, 0 moved; every changed predicate keeps its batch truth value.
  • Stream == batch, corpus confirmed-bar scan: 307 PASS / 0 FAIL / 5 refused of 312 (main: 264 / 43 / 5). Population sample: 34 / 0 / 86 refused of 120.
  • Gate sweep: SWEEP.

Verification

  • Focused tests 66/66 on this head, and 32/34 fail on main (two are controls). Hash coverage includes every new source state.
  • Two independent reviews, with all P1 findings fixed. The remaining follow-ups are documented: calendar-sealed interval closure on non-boundary bars, and one behaviour pin.
  • pineforge/verify and pineforge/parity are posted from the final tree.

🤖 Generated with Claude Code

from types import SimpleNamespace
from unittest.mock import Mock, patch

import native_confirmed_bar_stream_scan as scan
Comment on lines +42 to +45
self.assertTrue(' && leftover_flat_stop\n'
' && policy_script_bar_valid_\n'
' && !config_.calc_on_order_fills\n'
' && modeled_input()) {' in source)
Comment on lines +49 to +50
self.assertTrue(' if (last_bar_dual_entry_script_open_ms_ != context.script_bar_open_ms) {\n'
' last_bar_dual_entry_path_ = 0;' in source)
Correct the look-ahead description: Opening sibling-stop arbitration can inspect a complete script candle’s high and low in history. Live aggregation has only the children received so far; the opening script-bar seed likewise cannot contain its later children. The measured sibling-stop policy uses the supplied confirmed bar; the unavailable future-child opening seed remains documented.
@luisleo526
luisleo526 merged commit 7b59662 into main Oct 4, 2026
27 of 29 checks passed
@luisleo526
luisleo526 deleted the lv/stream-gates branch October 4, 2026 04:51
luisleo526 added a commit that referenced this pull request Oct 4, 2026
…s one

After the rebase onto #325 the entry sat at the top of Unreleased; it now
follows #325's entry, as the merge order asks. Wording unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 4, 2026
…s (--symbol-feeds) (#327)

* Install other symbols' bars for request.security in the Docker harness

docker/run_json.py --symbol-feeds (PINEFORGE_SYMBOL_FEEDS in the entrypoint)
reads a JSON index of other symbols' bars, keyed by the exact symbol string a
script passes to request.security and by timeframe, and installs each symbol's
catalog syminfo (tickerid as canonical, type, timezone, session, currency,
mintick) and feeds through strategy_set_symbol_facts and
strategy_set_symbol_feed, which engine and codegen have shipped since 1.0.0
but no release harness called. A feed is an OHLCV CSV like --ohlcv; each
bar's close is its open plus the timeframe unless a time_close column gives it.

Everything is checked before the first strategy state: timeframe spelling
(bare D/W/M/S folded, "4h" refused), one feed per timeframe, CSV columns and
numbers, increasing opens, closes at or before the next open, mintick, the
256-feed cap. A refusal, a library without the setters or a feed the engine
refuses is the structured {"engine","error"} line (harness exit 1, entrypoint
exit 4), like a rejected --syminfo. What was installed is recorded as
applied_runtime.symbol_feeds (facts, bar counts, first/last open, a value
hash), so such a run has its own fingerprint; without --symbol-feeds the
report is unchanged.

Stacked on harness/syminfo-mincontract (#322): it reuses that change's
structured-failure path in main().

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Address review of the symbol-feeds harness change

- Every refusal is the structured line, never a traceback: a feed that is not
  UTF-8 (or a CSV error), a time outside +-(2^53-1) ms or a monthly close
  outside the calendar, a mintick integer beyond binary64, a symbol or fact
  string with a lone surrogate, an index nested past the recursion limit.
- A header-only feed is installed without bars, as the engine documents it
  (its requests read na), instead of refused.
- An empty time_close cell falls back to open plus timeframe; a UTF-8 BOM is
  read; error line numbers are the CSV's own (blank lines counted).
- Install-time refusals carry the same "--symbol-feeds:" prefix.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Document the symbol-feed limits and that no feed is aggregated

The harness README gains a limits bullet (256 symbols and 256 feeds, the chart
as its own input with the engine's exact error, historical runs only, the cost
of a feed, request.security_lower_tf) and states that a feed must be at the
timeframe it serves: the engine aggregates only the chart's own input and the
C ABI has no call that aggregates bars, so a 1m feed serves neither a 240 nor a
D request. The CHANGELOG entry says the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Correct the symbol-feeds README on syminfo reads and aggregation

Review fixes. Inside the request syminfo.tickerid is always the key and
syminfo.ticker the key after its last ":"; the catalog tickerid is set as the
canonical fact, which no syminfo.* reads. The no-aggregation bullet now says
why (the engine looks a feed up by exact symbol and timeframe and installs its
bars as given) instead of claiming no C call aggregates. The missing-feed
sentence covers an index that lacks the requested symbol or timeframe, and the
refusal list is marked as examples ("timestamps that do not strictly
increase").

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Place the symbol-feeds CHANGELOG entry after the confirmed-bar streams one

After the rebase onto #325 the entry sat at the top of Unreleased; it now
follows #325's entry, as the merge order asks. Wording unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 4, 2026
Brings the branch up to date with main 4f2a475 (#325 7b59662, #327
4f2a475) so the release notes merge last, as the merge order now asks.

Conflicts resolved neutrally; the next commit does the editing:
- CHANGELOG.md: this branch's 1.1.0 section, with main's two new
  Unreleased bullets ("Confirmed-bar streams compute what the batch
  computes:" from #325, "Harness symbol feeds:" from #327) kept verbatim
  inside it; main's "## Unreleased" heading and its older copies of the
  bullets this branch already rewrote are not taken.
- README.md: #325's known-issue lines replace this branch's
  "Known issue (v1.0.0, v1.0.1, v1.1.0)" line from a80485c.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 4, 2026
… rule

#325 (confirmed-bar streams compute what the batch computes) and #327
(docker/run_json.py --symbol-feeds) merged before the release notes, so
their Unreleased bullets join the 1.1.0 section, rewritten and linked; no
Unreleased heading stays.

- CHANGELOG.md: the lead names the stream fix and, in the pairing
  sentence, codegen 1.1.0's request discovery (transpile_full()["requests"],
  codegen #164). #325's bullet says what a stream user sees; #327's bullet
  and its Report keys entry (applied_runtime.symbol_feeds, the
  --symbol-feeds: failure messages). A new State hashes section lists the
  four recipe additions since v1.0.1 (#315, #316, #319, #325; none removed,
  renamed or re-encoded; domain tags unchanged). #315's range-end
  equity-extreme change moves to the Pine adapter parity bullet as a
  behaviour fix. The #319 bullet named the broker-state hash; the change is
  in the native continuation hash, which the broker-state hash folds. The
  known-issue bullet now says it is fixed in 1.1.0 (#325). Migration gains
  a line on stream results and hashes.
- README.md: the known issue is fixed in 1.1.0 (#325); the v1.1.0 Releases
  line gains the stream fix, request discovery and the harness's
  multi-symbol request.security feeds. Fact markers rendered from the facts
  export that maps releases[1.1.0] to
  pineforge-parity-baseline-20261004-engine-7b596622 (7,951 excellent / 38
  strong of 7,989); the active scoreboard moves to the same baseline, here
  and in docs/pages/contributing-llm.md.
- docs/pages/streaming.md: the known issue affected v1.0.0 and v1.0.1 and
  is fixed in 1.1.0.
- docs/pages/public-contract.md: the state-hash rule, amended by the
  owner's ruling for 1.1.0. It covers the recipe only: a minor release may
  add a hashed field (domain tags unchanged, the added field changing no
  trade or report, disclosed under State hashes); removing, renaming or
  re-encoding a hashed field is a new epoch and a major release; behaviour
  changes follow the normal release and parity rules. The first uses are
  #315 and #316. The semver rule is unchanged.

Replaces text introduced by 7b59662 (#325): CHANGELOG.md's
"Confirmed-bar streams compute what the batch computes:" bullet, and the
"Fixed on main (#325); included in the next release." lines of README.md
and streaming.md; by 4f2a475 (#327): CHANGELOG.md's "Harness symbol
feeds:" bullet; by d093560 (#290): public-contract.md's "State-hash
values are stable within the epoch." and "a new recipe is a new epoch and
a major release"; by a80485c (this branch's first commit): the "Known
issue, as in v1.0.0 and v1.0.1" bullet, streaming.md's "(v1.0.0, v1.0.1,
v1.1.0)", the #319 bullet's "A tick stream's broker-state hash folds the
new accumulator state only where it adds information, so the existing
tick, bar and batch hash pins hold.", the Report keys failure-output
bullet and the v1.1.0 Releases line; by 66be240 and 83d64dc: the lead's
state-hash sentences ("Three changes also change what the hashes fold");
and, as git attributes them through this branch's merge of main, by
e59c09f: README.md's v1.1.0 Releases line and streaming.md's
"(v1.0.0, v1.0.1, v1.1.0)".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 4, 2026
* docs: release notes and version scoping for v1.1.0

v1.1.0 is a minor release: <pineforge/pineforge.h> declares six more
functions, the checked settings calls that a strategy generated by
pineforge-codegen 1.1.0 exports (#317). It pairs with pineforge-codegen
1.1.0. PF_ABI_VERSION stays 4 (pineforge.h only gains declarations;
native_c_api.h and the native C++ host headers are unchanged) and the
script ABI epoch stays engine_script_run_v19.

- CHANGELOG.md: "## Unreleased" becomes "## 1.1.0 — 2026-10-04": a lead
  paragraph, one bullet per change a user can observe since v1.0.1
  (#312-#324), each linking its PR, a Report keys subsection
  (docker/run_json.py v1.0.1 -> v1.1.0) and a Migration subsection. The
  three Unreleased bullets stay; the lot-grid bullet now gives both
  failure messages.
- README.md: pineforge-codegen==1.1.0, the v1.1.0 tarballs, the pairing
  sentence, the stream known issue's scope, the benchmark refresh line,
  the release scoreboard sentence (releases[1.1.0] markers) and a v1.1.0
  line in Releases. The public corpus sweep's "311 excellent + 1 declared
  anomaly" is scoped to the v1.0.1 library it was measured on. Facts
  rendered from the facts file that adds releases[1.1.0]; the active
  scoreboard markers move to baseline
  pineforge-parity-baseline-20261003-engine-dbd17b38.
- docs/pages/install.md: the v1.1.0 tarballs and the hub tag
  engine1.1.0-codegen1.1.0.
- docs/pages/public-contract.md: 1.1.0's release line, engine v1.1.0 with
  codegen 1.1.0 among the supported pairs, and the four C-boundary rows
  that were planned for 1.1.0 are not in it.
- docs/pages/streaming.md: the known issue's scope includes v1.1.0.

Lines that state what an earlier version shipped or did stay.

Replaces text introduced by dc74e63 (#318): README.md's "Release 1.0.1
still grades ... until the next release" sentence, now 1.1.0's; by
d093560 (#290): public-contract.md's "Four of its rows are planned for
1.1.0"; by 783bb39 (#322): CHANGELOG.md's "The engine library is
unchanged."; by dbd17b3 (#320): CHANGELOG.md's "The engine is
unchanged."; by b2a578c (#300): README.md's "this repository's sweep:
311 excellent + 1 declared anomaly"; by 271d687 (#311): README.md's
"its releases 1.0.0 and 1.0.1 do".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs: state the 1.1.0 hash folds and scope two runner sentences

The 1.1.0 lead now says how state-hash values change while the epoch and
the domain tags stay: a run whose fills an adapter change alters hashes
to new values, and three changes fold more state, so a run can hash
differently from 1.0.1 with byte-identical trades: #315 (an unbatched
same-bar entry request and placement; the stream report's terminal
re-mark leaves the hashed extremes; witness row Stream/0/1 re-pinned),
#316 (an opening stop's next waypoint; Random44/0/0 and /1 re-pinned)
and #319 (tick-volume state, every existing pin kept).

In the runner tooling bullet, "No versioned engine PF_API export ...
changes" and "Plugin-free ledger identity bytes remain unchanged" read
as release-wide; they now speak for #320 itself and for a strategy
without the checked settings calls. The Report keys intro points at the
report schema page. The terminal-quote bullet names the chart feed that
PINEFORGE_RUN_REPORT_CHART_QUOTE points at, and two runner bullet
headings are re-wrapped.

Replaces text introduced by a80485c (this branch's first commit): the
lead's "A run whose fills one of the Pine adapter changes below alters
hashes to new values, and [#316] adds one placement field to the
adapter's hashed state, for which two witnesses with byte-identical
trades were re-pinned.", the Report keys intro and the terminal-quote
harness clause; and text introduced by dbd17b3 (#320): "No versioned
engine PF_API export, native C++ surface, script ABI epoch or engine
behavior changes." and "Plugin-free ledger identity bytes remain
unchanged".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs: #315's range-end change folds less, not more

The 1.1.0 lead said three changes "fold more state"; #315's range-end
change instead stops folding a stream report's terminal re-mark into the
hashed extremes. The sentence now says the three changes change what the
hashes fold, and names Stream/0/1 as a re-pinned witness row without
claiming it is #315's only one.

Replaces text introduced by 66be240 (this branch's second commit): "Three
changes also fold more state" and "re-pinning one witness row
(`Stream/0/1`, whose recorded-row digest alone moves)".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs: fold #325 and #327 into 1.1.0, add State hashes, amend the hash rule

#325 (confirmed-bar streams compute what the batch computes) and #327
(docker/run_json.py --symbol-feeds) merged before the release notes, so
their Unreleased bullets join the 1.1.0 section, rewritten and linked; no
Unreleased heading stays.

- CHANGELOG.md: the lead names the stream fix and, in the pairing
  sentence, codegen 1.1.0's request discovery (transpile_full()["requests"],
  codegen #164). #325's bullet says what a stream user sees; #327's bullet
  and its Report keys entry (applied_runtime.symbol_feeds, the
  --symbol-feeds: failure messages). A new State hashes section lists the
  four recipe additions since v1.0.1 (#315, #316, #319, #325; none removed,
  renamed or re-encoded; domain tags unchanged). #315's range-end
  equity-extreme change moves to the Pine adapter parity bullet as a
  behaviour fix. The #319 bullet named the broker-state hash; the change is
  in the native continuation hash, which the broker-state hash folds. The
  known-issue bullet now says it is fixed in 1.1.0 (#325). Migration gains
  a line on stream results and hashes.
- README.md: the known issue is fixed in 1.1.0 (#325); the v1.1.0 Releases
  line gains the stream fix, request discovery and the harness's
  multi-symbol request.security feeds. Fact markers rendered from the facts
  export that maps releases[1.1.0] to
  pineforge-parity-baseline-20261004-engine-7b596622 (7,951 excellent / 38
  strong of 7,989); the active scoreboard moves to the same baseline, here
  and in docs/pages/contributing-llm.md.
- docs/pages/streaming.md: the known issue affected v1.0.0 and v1.0.1 and
  is fixed in 1.1.0.
- docs/pages/public-contract.md: the state-hash rule, amended by the
  owner's ruling for 1.1.0. It covers the recipe only: a minor release may
  add a hashed field (domain tags unchanged, the added field changing no
  trade or report, disclosed under State hashes); removing, renaming or
  re-encoding a hashed field is a new epoch and a major release; behaviour
  changes follow the normal release and parity rules. The first uses are
  #315 and #316. The semver rule is unchanged.

Replaces text introduced by 7b59662 (#325): CHANGELOG.md's
"Confirmed-bar streams compute what the batch computes:" bullet, and the
"Fixed on main (#325); included in the next release." lines of README.md
and streaming.md; by 4f2a475 (#327): CHANGELOG.md's "Harness symbol
feeds:" bullet; by d093560 (#290): public-contract.md's "State-hash
values are stable within the epoch." and "a new recipe is a new epoch and
a major release"; by a80485c (this branch's first commit): the "Known
issue, as in v1.0.0 and v1.0.1" bullet, streaming.md's "(v1.0.0, v1.0.1,
v1.1.0)", the #319 bullet's "A tick stream's broker-state hash folds the
new accumulator state only where it adds information, so the existing
tick, bar and batch hash pins hold.", the Report keys failure-output
bullet and the v1.1.0 Releases line; by 66be240 and 83d64dc: the lead's
state-hash sentences ("Three changes also change what the hashes fold");
and, as git attributes them through this branch's merge of main, by
e59c09f: README.md's v1.1.0 Releases line and streaming.md's
"(v1.0.0, v1.0.1, v1.1.0)".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 5, 2026
* docs: release notes and version scoping for v1.2.0

CHANGELOG: the Unreleased section becomes 1.2.0 (2026-10-05), a minor
release that adds the compiled execution-capabilities extension to the C
ABI (PF_CAPABILITIES_API_VERSION, strategy_capabilities_api_version,
strategy_capabilities_receipt). One bullet per change since v1.1.0 with
its PR (#328, #330, #331, #332, #333, #334, #335, #336, #337, #338 and the
session-clock change, written #VCP until its PR number exists), the
recipe additions under State hashes (#328, #330, #335; domain tags and
epoch unchanged), Report keys (docker/run_json.py is byte-identical to
v1.1.0's, so no key changes) and Migration.

Version scoping: README, install and public-contract pages name v1.2.0
and codegen 1.2.0; "since the next release" and "fixed on main" in the
capabilities guide and the runner guide become 1.2.0 and 1.1.0 (#325).
Lines that state what an earlier release shipped stay.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs: 1.2.0's grades from the facts tokens, #339, and codegen's license

- CHANGELOG: the session-clock change is #339 (merged as 52292db), in place
  of the #VCP token of 3082fbd; the "What a script author sees" bullet of
  3082fbd now states release 1.2.0's grades (registry baseline
  pineforge-parity-baseline-20261005-engine-52292db9: 7,970 excellent / 19
  strong of 7,989, none below strong) instead of the main scoreboard at
  6b77f06.
- README: the release sentence names 1.2.0 and renders releases[1.2.0], and
  the main scoreboard renders the same baseline, from pineforge-release
  facts/facts.json at 0c2f6838 (the facts export for the 1.2.0 release).
- README, LEGAL.md, docs/pages/index.md: pineforge-codegen is under the
  PineForge Source License 1.1 from its release 1.2.0; releases up to 1.1.0
  keep the PolyForm Noncommercial terms they shipped with.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant