Skip to content

Commit 7f807b9

Browse files
luisleo526claude
andauthored
Another symbol's bars reach a run as a feed: kernel instrument feeds + Pine foreign request sites (XSYM-D) (#292)
* XSYM-D: another symbol is a feed -- kernel instrument feeds merged by interval, and the Pine adapter's request sites of other symbols Kernel (ADR-0001 rule 3, opt-in). NativeRunSpec::instrument_feeds holds other instruments' own bars, keyed by (instrument, timeframe), each bar with its own close (close_ms) and optional named columns; a NativeSeriesSource::InstrumentFeed series names one. On each accepted input the series is handed, in feed order, every bar not yet handed over that has closed by the input's NativeInterval::last_traded_close_ms (lookahead off) or opened by its open_ms (lookahead on); gaps is the subscription contract's; bars before the first input ride on it. No calendar is guessed for the instrument; the chart side reads the input's calendar interval (resolved from the label on a raw-label partition, whose inputs carry no duration). With no feed installed every digest, delivery and run is what it was. Streams refuse a spec that installs one. Nine named refusals (NativeRunSpecError/Field, with C twins so the translation stays exhaustive); C has no route (1.0 boundary row). Adapter. register_security_eval(sec_id, symbol, tf, input_tf, lookahead_on, gaps_on, ignore_invalid_symbol) registers a site of another symbol; the host projects its installed feeds as the spec's instrument feeds, declares one InstrumentFeed series per site after its same-symbol sites (which keep their route), and runs the payload on every bar handed over in the requested context (history and TA over the context's bars, bar_index, time_close and syminfo.* the context's). gaps_on clears at the bar open, before the COOF checkpoint. An invalid symbol reads na under ignore_invalid_symbol and fails closed without it; a missing feed fails closed naming symbol and timeframe; an aggregated chart is refused. Recorded request series: key -> chart-bar open -> value, na where the tape has no row. C ABI: strategy_set_symbol_feed / _feed_column / strategy_set_symbol_facts / strategy_set_recorded_series, each behind its own PINEFORGE_HAS_*_V1 probe, reached through BacktestEngine virtuals whose kernel base answers false; a source host refuses stream_begin while any holds data, and check_c_abi_runtime.py pins that each declaration says so. Tests on synthetic bars only: test_native_instrument_feed (kernel, the report's pinned cases), test_native_instrument_feed_twin (kernel/adapter twin), test_symbol_request_data (C ABI, recorded series, stream refusal). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * XSYM-D: run_strategy.py installs a probe's pinned request data from PINEFORGE_REQUESTS_ROOT The environment contract of workflow docs/xsym-requests.md: when PINEFORGE_REQUESTS_ROOT is set and <root>/<basename of the strategy dir>/ requests.json exists, the harness holds the manifest to pineforge-probe-requests/v1 (the refusals of the workflow's validateProbeRequests) and to this probe, verifies every file it names under the probe's own files/ against its sha256 (a feed against its byte count), parses the feeds (an empty or NaN volume or extra value is NaN), the recorded tapes and the symbol facts, and installs them through strategy_set_symbol_facts / _feed / _feed_column / strategy_set_recorded_series before the run. It reads nothing else: never another slug's directory, never the case-wide PINEFORGE_PINE_LIBRARIES. The run provenance records each feed's (key, sha, bars), each tape's (key, sha, rows) and the manifest's sha, as it does for the aux and native security feeds. A malformed manifest stops the run by name before any engine call; a library without the setters and the docker runner are refused. Unset, or no manifest for the probe: the engine is handed exactly what it always was. The frozen verifier names the probe's build directory BUILD / slug (pineforge-lab 3bac0b7b scripts/verify-engine-local.py:1695) and runs this script on it (:1593-1596, :1614-1617) through _run_timed (:190-193), whose subprocess.run passes no env=, so the runner's environment reaches it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * XSYM-D docs: "only the run's own symbol is addressable" becomes the instrument-feed contract native-engine.md: the field list, the Limits paragraph and the stream-limits bullet state the new contract; a new section "Instrument feeds: another symbol's bars" gives the struct, the delivery rule, the validation table, the stream refusal and the Pine adapter's use (the symbol-key overload, the payload in the requested context, ignore_invalid_symbol, the fail-closed refusals, recorded request series, the run_strategy.py environment contract); the 1.0 C boundary table has the "instrument feeds" row its checker pins. ADR-0001: the rule-2 ruling row "Wave XSYM-D" (mechanism: another venue is driven by changing data only; knob: none -- lookahead and gaps are already the subscription's delivery rules), calibrated by XSYM-DESIGN's 203/203 merge pins and this lane's two witness reproductions; the HTF paragraph, the run-spec paragraph and the adapter's section name the new surface. mtf.md, coverage.md and README: the other-symbol surface, the recorded fundamentals store, the four C setters; the runtime export counts move 58 -> 62 (66 -> 70 declared, 109 -> 113 in total). Sentences this commit removes, each superseded by the contract it states: - ADR-0001 (5691220, lane L14-C): "request.security-style series are reachable from a bare host, as declared series of the run's own symbol" -- another symbol's bars are reachable now too, and the paragraph says so. - coverage.md (cc4a56e): "Runtime owns the security state machine, ratio / calendar aggregation, lookahead / gaps semantics, lower-TF emulation, and per-security diagnostics", "The runtime owns same-symbol security computation" and "Why not done yet: Most strategy logic in our test corpus relies on the chart symbol's bars + indicators" -- the request.security row and its section state the other-symbol surface instead. - coverage.md (9507d8f): "Would need a parallel data-ingestion path so the user can supply a CSV / Parquet of fundamentals", "Future story: Fundamentals and economic indicators would land as a generic auxiliary timeline feature" and "Fundamentals- and macro-driven strategies are a meaningful but smaller user segment" -- the recorded request series store and the pinned requests manifest are that ingestion path. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * XSYM-D: doc anchors re-anchored on 1e360bd; ctest floors for the lane's four rows The rebase onto 1e360bd (INT27 with its round-2 fix) shifted cited lines: reanchor_doc_citations.py --ref 1e360bd plus check_doc_anchors.py --fix and four hand fixes bring every anchor back (1725/1725; the diff changes digits only). Floors, counted with ctest -N on the lane's tree under each ci_verify profile's cache definitions: kernel 292 registered, 291 run (+2: test_native_instrument_feed, test_run_strategy_requests; the WebSocket row still skips) release 720 (+4: the two above, test_native_instrument_feed_twin, test_symbol_request_data) debug/sanitizers/native registration 701/701/710 (+4 each) KERNEL_MIN_TESTS 289 -> 291, RELEASE_MIN_TESTS 716 -> 720, EXCLUDED_REGISTERED_MIN 697/697/706 -> 701/701/710; docs/ci.md and the floor citations follow. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * XSYM-D review fixes: an empty tape or feed is installed (na, not unfed); setters refuse a run in progress; the literal input_tf trap deleted An independent review of 1e360bd..80236d5a found, and this commit fixes: - The workflow keeps a request that was na on every chart bar as a tape with a header only (campaign/tv/xsym/decode.mjs keeps non-na rows), and pins it. strategy_set_recorded_series(key, ..., 0) erased the key, so the first read failed the run as "no recorded series is installed". n == 0 now installs a series without rows: every read is na. strategy_set_symbol_feed(..., 0) likewise installs a feed without bars, whose sites read na, instead of leaving the symbol unfed. - The four setters answer -1 while a run is in progress (a stream's warmup or realtime leg included), naming it: the source host's mutation guard is inert, so a mid-stream install changed what the run read and its digest. - strategy_set_symbol_feed_column judges the column name the kernel's way at install (a name that is not UTF-8 was accepted, then refused at begin). - register_security_eval(id, "SYM", "tf", "literal", bool[, bool]) bound the same-symbol overload (const char* -> bool outranks std::string) and read the symbol as the timeframe. That spelling is deleted; generated code passes input_tf_ (a std::string) and is unaffected. - run_strategy.py refuses by name a manifest the schema cannot walk (JSON nested past the recursion limit, a mintick past a double, an unreadable file) and a stamp an int64 cannot hold, which ctypes would wrap silently. - abi-stability.md records the four BacktestEngine virtuals added inside v19 (as N5 did inside v18); mtf.md and native-engine.md state the empty installs, the in-run refusal and that the generated clear_security() also clears a gaps_on payload's own history (a codegen limit). Tests: test_symbol_request_data 77 -> 119 checks (empty tape, empty feed, column name, in-run refusal, the deleted spelling as static_asserts); test_native_instrument_feed 10543 -> 12375 (a raw label outside the session reads the calendar's zero-traded slot); test_run_strategy_requests 13 -> 15. Fail-before on 80236d5a's headers and library: the static_asserts fail to compile; 15 runtime failures; the harness raises OverflowError. Sentences of 4076d7d this commit rewrites: mtf.md's and native-engine.md's "gaps_on reads na on a chart bar that received no new bar" now also states that the generated clear_security() resets the payload's own history, and mtf.md's "Historical runs only: a stream refuses an installed symbol feed" now also states the in-run refusal. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * XSYM-D: the symbol-data group comment is the first probe's own doc (detached-comments census 8 -> 0) The preflight's detached-comments stage (measure_detached_comments.py --check-ceiling, ceiling 0) counted the eight-line group comment above the four PINEFORGE_HAS_…_V1 probes as detached: a second block in the gap above the first probe's own doc. Its text now opens that doc, so the census of the kernel compile closure is 0 again; the lines keep their numbers. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 1e360bd commit 7f807b9

42 files changed

Lines changed: 5326 additions & 440 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -152,8 +152,8 @@ What each gate refuses:
152152
### The floors
153153

154154
`ci_verify.py` counts the CTest rows that actually **ran** and fails below a
155-
floor — `KERNEL_MIN_TESTS` ci_verify.py:274 and `RELEASE_MIN_TESTS`
156-
ci_verify.py:429.
155+
floor — `KERNEL_MIN_TESTS` ci_verify.py:284 and `RELEASE_MIN_TESTS`
156+
ci_verify.py:447.
157157
A deleted or silently skipped row is a failure, not a quieter run. If your
158158
change adds rows, raise the floor in the same commit and say by how much; if it
159159
legitimately removes one, lower it deliberately and say why. `--min-tests`

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3535
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) uses PolyForm Noncommercial terms with additional personal-trading permission; commercial use requires a separate license. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3636
- **Fast.** In-process, no interpreter: median **15× faster than PyneCore** on 196 timed strategies (a median 603k bars/s per strategy with the bar magnifier on). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3737
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
38-
- **Yours to embed.** 109 `extern "C"` functions across two headers — 66 compiled-strategy declarations and 43 native-host declarations — append-only ABI. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
38+
- **Yours to embed.** 113 `extern "C"` functions across two headers — 70 compiled-strategy declarations and 43 native-host declarations — append-only ABI. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3939

4040
---
4141

@@ -436,9 +436,9 @@ API, the script ABI epoch and the pairing with codegen — is the
436436
[public contract](docs/pages/public-contract.md); what a 0.x user must act on
437437
is in [CHANGELOG.md](CHANGELOG.md).
438438

439-
A built strategy `.so` exposes 66 compiled-strategy `PF_API` declarations
440-
(58 runtime implementations plus eight generated exports) plus 43 native-host
441-
declarations: 109 `PF_API` exports in total. `nm -gU` also shows libc++'s
439+
A built strategy `.so` exposes 70 compiled-strategy `PF_API` declarations
440+
(62 runtime implementations plus eight generated exports) plus 43 native-host
441+
declarations: 113 `PF_API` exports in total. `nm -gU` also shows libc++'s
442442
`std::piecewise_construct`; no project-internal C++ symbol is exported. The two
443443
inventories are pinned by `scripts/check_c_abi_runtime.py`:
444444

@@ -457,6 +457,7 @@ inventories are pinned by `scripts/check_c_abi_runtime.py`:
457457
| `strategy_set_chart_timezone` / `strategy_set_syminfo_timezone` / `strategy_set_syminfo_session` | Chart and exchange time |
458458
| `strategy_set_syminfo_mintick` / `_pointvalue` / `_metadata` / `_type` / `_string` | Symbol tick size, point value, numeric metadata, instrument class, string members |
459459
| `strategy_set_native_security_feed` / `strategy_set_aux_security_feed` | Feed `request.security()` from a native higher-timeframe series / an auxiliary bar-aligned feed |
460+
| `strategy_set_symbol_feed` / `_feed_column` / `strategy_set_symbol_facts` / `strategy_set_recorded_series` | Another symbol's data for `request.security()` of that symbol: its own bars (each with its close) and named columns, its `syminfo.*` facts, and recorded request values per chart bar (each behind its own `PINEFORGE_HAS_…_V1` probe; historical runs only) |
460461
| `strategy_set_account_currency_fx_series` | Effective-time quote-to-account FX |
461462
| `strategy_get_last_error` | The latest runtime error |
462463
| `pf_version_get` / `pf_version_string` / `pf_abi_version` | Runtime version, version string, struct-layout version (`PF_ABI_VERSION == 4`) |

‎docs/adr/0001-kernel-adapter-boundary.md‎

Lines changed: 93 additions & 76 deletions
Large diffs are not rendered by default.

‎docs/ci.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -102,11 +102,11 @@ requires the rows that actually ran to equal the second count, and records
102102
`registered`, `labelled = registered - selected`, and `ran` in
103103
`ctest-exclusion.log` and `ci-summary.json`. A skip, missing executable, lost
104104
registration below the PR registration floor, unreadable enumeration, or
105-
missing label fails. The PR registration floors at INT27 are 697 for Debug
106-
and sanitizers and 706 for native (653 and 662 at the INT24 base, plus wave G's
107-
six rows, wave H's twenty-four, INT26's own tape row and INT27's thirteen). The
108-
full-run release and kernel floors are 716 and 289 rows that ran; full runs do
109-
not exclude a label.
105+
missing label fails. The PR registration floors after lane XSYM-D are 701 for
106+
Debug and sanitizers and 710 for native (653 and 662 at the INT24 base, plus
107+
wave G's six rows, wave H's twenty-four, INT26's own tape row, INT27's thirteen
108+
and XSYM-D's four). The full-run release and kernel floors are 720 and 291 rows
109+
that ran; full runs do not exclude a label.
110110

111111
Preflight also runs the detached-comment census of the kernel compile closure
112112
(`detached-comments`: `scripts/measure_detached_comments.py --check-ceiling`)

‎docs/coverage.md‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@
4242
| `ta.`* | Broad runtime support | 59 official Pine v6 `ta.`* functions plus 8 official `ta.`* series variables backed by stateful runtime classes, and a free `pivot_point_levels(...)`. Stateful classes expose both `compute(...)` (advance state) and `recompute(...)` (re-run on the same bar without permanently advancing history). |
4343
| `math.`* | Narrow runtime backing | Runtime owns only deterministic `pine_random(...)` and rolling `math::Sum`; everything else is left to consumer-emitted code. |
4444
| `str.`* | Narrow runtime backing | Runtime owns `pine_str_format`, `pine_str_format_time`, `pine_str_match`, `pine_str_split`, `pine_str_tostring`. |
45-
| `request.security()` | Partial | Runtime owns the security state machine, ratio / calendar aggregation, lookahead / gaps semantics, lower-TF emulation, and per-security diagnostics. |
45+
| `request.security()` | Partial | Runtime owns the security state machine, ratio / calendar aggregation, lookahead / gaps semantics, lower-TF emulation, per-security diagnostics, and (lane XSYM-D) the merge of another symbol's installed bars; codegen still refuses another symbol until its lowering lands. |
4646
| Bar magnifier | Supported | OHLC-path sampling with 6 distribution modes plus optional volume-weighted sample density. |
4747
| Time / session / timezone | Supported | `pine_time` / `pine_time_close` with session filtering and a mutex-guarded `tz_util::ScopedTimezone`. |
4848
| Timeframe parsing | Supported | `tf_to_seconds`, `tf_ratio`, `tf_change`, `detect_timeframe`, calendar boundary detection, `TimeframeAggregator` (passthrough / ratio / calendar). |
@@ -60,7 +60,7 @@
6060
## Public C ABI
6161

6262
`<pineforge/pineforge.h>` is the **single canonical consumer header**. It has
63-
exactly 66 public `PF_API` declarations: 58 runtime implementations and eight
63+
exactly 70 public `PF_API` declarations: 62 runtime implementations and eight
6464
per-strategy generated exports. Every compiled PineForge strategy `.so` exports
6565
that public set. The historical 28-symbol module sentence was not a current
6666
module inventory; the grouped table below is a guide, not the count:
@@ -124,7 +124,7 @@ single `.hpp`):
124124

125125
| Module | Header | Source | Pine-facing role |
126126
| ------------------ | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
127-
| Public C ABI | `pineforge.h` | `c_abi.cpp` (+ layout `static_assert`s) | 66 public `PF_API` declarations: 58 runtime implementations plus eight per-strategy generated exports. `strategy_configure_native_fx_curve_v1` stages the additive native FX curve. |
127+
| Public C ABI | `pineforge.h` | `c_abi.cpp` (+ layout `static_assert`s) | 70 public `PF_API` declarations: 62 runtime implementations plus eight per-strategy generated exports. `strategy_configure_native_fx_curve_v1` stages the additive native FX curve. |
128128
| Engine | `engine.hpp` | `engine_run.cpp`, `engine_stream.cpp`, `engine_execution.cpp`, `engine_orders.cpp`, `engine_path_resolve.cpp`, `engine_trade_accessors.cpp`, `engine_security.cpp`, `engine_lower_tf.cpp`, `engine_report.cpp`, `native_execution_consumer.cpp` | One-shot and continuous lifecycle, native request matching/settlement, orders, reports, inputs / syminfo, magnifier, TF aggregation, and `request.security` plumbing. |
129129
| Engine internals | `engine_internal.hpp` | (private cross-TU header) | `pineforge::internal::`* types and helpers shared between engine `.cpp` partitions; not part of the public ABI. |
130130
| Technical analysis | `ta.hpp` | `ta_moving_averages.cpp`, `ta_oscillators.cpp`, `ta_volatility_trend.cpp`, `ta_extremes_volume.cpp`, `ta_misc.cpp` | Official `ta.`* functions and series variables backed by stateful runtime classes with `compute` / `recompute`, plus `pivot_point_levels(...)` free function. |
@@ -551,7 +551,8 @@ backing.
551551

552552
## `request.security()`
553553

554-
The runtime owns same-symbol security computation. Per-call state lives
554+
The runtime owns same-symbol security computation, and since lane XSYM-D
555+
the merge of another symbol's installed bars (below). Per-call state lives
555556
in `SecurityEvalState`:
556557

557558
```cpp
@@ -599,6 +600,21 @@ and requested timeframes are fixed intraday minute strings (no `D / W / M / S` s
599600
when `lookahead_on` or `gaps_on` is set — emulation is
600601
`lookahead_off / gaps_off` only.
601602
603+
**Another symbol.** A site of another symbol registers with
604+
`register_security_eval(sec_id, symbol, requested_tf, input_tf, lookahead_on,
605+
gaps_on, ignore_invalid_symbol)` (`PINEFORGE_HAS_SYMBOL_SECURITY_EVAL_V1`) and
606+
reads that symbol's own bars, installed before the run through
607+
`strategy_set_symbol_feed` / `_feed_column` / `strategy_set_symbol_facts`. The
608+
kernel hands the site every bar of that feed that has closed by the chart
609+
bar's close (`lookahead_off`) or opened by its open (`lookahead_on`), in
610+
order, each with the close the feed gives it
611+
(`NativeRunSpec::instrument_feeds`); the source host runs the payload on each
612+
in the requested context -- its own history, `bar_index`, `time_close` and
613+
`syminfo.*` -- and fails the run closed, naming the symbol and the timeframe,
614+
when no feed is installed. Codegen still refuses a symbol that is not the
615+
chart's until its lowering onto this surface lands (lane XSYM-E); see
616+
@ref native_engine, "Instrument feeds: another symbol's bars".
617+
602618
`request.security_lower_tf(...)` is supported for same-symbol lower
603619
timeframes that satisfy the same emulation constraints. It returns an
604620
array whose elements are ordered earliest-to-latest within the chart bar,
@@ -888,11 +904,12 @@ loudly via a generic `request.`* catch-all, so user code never silently
888904
falls through to broken codegen.
889905
890906
- **Feasibility:**
891-
- `request.financial / dividends / earnings / splits / currency_rate / economic`: *Feasible — needs aux data*. Would need a parallel data-ingestion path so the user can supply a CSV / Parquet of fundamentals, corporate actions, or macroeconomic series; the runtime would then look up the right slice by `bar.timestamp`.
907+
- `request.financial / dividends / earnings / splits`: *Feasible — the runtime half exists*. The source host keeps recorded request series (`strategy_set_recorded_series`, lane XSYM-D): a request key maps each chart bar's open time to the value TradingView returned on it, recorded under the call's own `gaps` and `lookahead`, and `recorded_series_value(key)` reads it per chart bar, na where the tape has no row. Recording per chart bar, rather than re-deriving TradingView's report-time and fiscal-period rules from an event list, keeps the value exact by construction. What is missing is codegen's lowering onto the store and the tapes themselves, which the campaign pins per probe (`scripts/run_strategy.py` installs them from `PINEFORGE_REQUESTS_ROOT`).
908+
- `request.currency_rate / economic`: *Feasible — needs aux data*. Would need the same kind of pinned series, which no lane records yet.
892909
- `request.seed`: *Out of scope structurally*. TradingView seeds are user-published time series hosted on TV's infrastructure; PineForge has no equivalent registry.
893910
- `request.quandl`: *Out of scope by design*. Deprecated upstream; not worth implementing.
894-
- **Future story:** Fundamentals and economic indicators would land as a generic "auxiliary timeline" feature: pass `aux_data={"earnings": df, "us_gdp": df}` to the runner, exposed to Pine via `request.financial` / `request.economic`-shaped accessors that read from the aux table.
895-
- **Why not done yet:** Most strategy logic in our test corpus relies on the chart symbol's bars + indicators. Fundamentals- and macro-driven strategies are a meaningful but smaller user segment; we have not built the aux-data ingestion contract.
911+
- **Future story:** codegen lowers `request.earnings` / `dividends` / `splits` / `financial` onto the recorded series by their request key (`<fn>|<symbol>|<field-or-id>|<period-or->|gaps_<on|off>|lookahead_<on|off>`), and a request with nothing recorded refuses on first read.
912+
- **Why not done yet:** the engine store landed first (lane XSYM-D); the codegen lowering is lane XSYM-E.
896913
897914
#### `barmerge.lookahead_on` for lower-TF emulation
898915

0 commit comments

Comments
 (0)