Skip to content

Commit 4f2a475

Browse files
luisleo526claude
andauthored
Install other symbols' bars for request.security in the Docker harness (--symbol-feeds) (#327)
* Install other symbols' bars for request.security in the Docker harness docker/run_json.py --symbol-feeds (PINEFORGE_SYMBOL_FEEDS in the entrypoint) reads a JSON index of other symbols' bars, keyed by the exact symbol string a script passes to request.security and by timeframe, and installs each symbol's catalog syminfo (tickerid as canonical, type, timezone, session, currency, mintick) and feeds through strategy_set_symbol_facts and strategy_set_symbol_feed, which engine and codegen have shipped since 1.0.0 but no release harness called. A feed is an OHLCV CSV like --ohlcv; each bar's close is its open plus the timeframe unless a time_close column gives it. Everything is checked before the first strategy state: timeframe spelling (bare D/W/M/S folded, "4h" refused), one feed per timeframe, CSV columns and numbers, increasing opens, closes at or before the next open, mintick, the 256-feed cap. A refusal, a library without the setters or a feed the engine refuses is the structured {"engine","error"} line (harness exit 1, entrypoint exit 4), like a rejected --syminfo. What was installed is recorded as applied_runtime.symbol_feeds (facts, bar counts, first/last open, a value hash), so such a run has its own fingerprint; without --symbol-feeds the report is unchanged. Stacked on harness/syminfo-mincontract (#322): it reuses that change's structured-failure path in main(). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Address review of the symbol-feeds harness change - Every refusal is the structured line, never a traceback: a feed that is not UTF-8 (or a CSV error), a time outside +-(2^53-1) ms or a monthly close outside the calendar, a mintick integer beyond binary64, a symbol or fact string with a lone surrogate, an index nested past the recursion limit. - A header-only feed is installed without bars, as the engine documents it (its requests read na), instead of refused. - An empty time_close cell falls back to open plus timeframe; a UTF-8 BOM is read; error line numbers are the CSV's own (blank lines counted). - Install-time refusals carry the same "--symbol-feeds:" prefix. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Document the symbol-feed limits and that no feed is aggregated The harness README gains a limits bullet (256 symbols and 256 feeds, the chart as its own input with the engine's exact error, historical runs only, the cost of a feed, request.security_lower_tf) and states that a feed must be at the timeframe it serves: the engine aggregates only the chart's own input and the C ABI has no call that aggregates bars, so a 1m feed serves neither a 240 nor a D request. The CHANGELOG entry says the same. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Correct the symbol-feeds README on syminfo reads and aggregation Review fixes. Inside the request syminfo.tickerid is always the key and syminfo.ticker the key after its last ":"; the catalog tickerid is set as the canonical fact, which no syminfo.* reads. The no-aggregation bullet now says why (the engine looks a feed up by exact symbol and timeframe and installs its bars as given) instead of claiming no C call aggregates. The missing-feed sentence covers an index that lacks the requested symbol or timeframe, and the refusal list is marked as examples ("timestamps that do not strictly increase"). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Place the symbol-feeds CHANGELOG entry after the confirmed-bar streams one After the rebase onto #325 the entry sat at the top of Unreleased; it now follows #325's entry, as the merge order asks. Wording unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 7b59662 commit 4f2a475

5 files changed

Lines changed: 847 additions & 4 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,19 @@ version number follows semantic versioning over the surfaces the
3131
unchanged. The streaming guide gains a "Backtest vs live" section listing what a
3232
backtest can see that a live run cannot.
3333

34+
- **Harness symbol feeds:** `docker/run_json.py --symbol-feeds` (`PINEFORGE_SYMBOL_FEEDS`)
35+
installs other symbols' bars for `request.security` on another symbol: a JSON
36+
index keyed by the exact symbol string the script passes and by timeframe, each
37+
feed an OHLCV CSV plus the symbol's catalog `syminfo`, through
38+
`strategy_set_symbol_facts` / `strategy_set_symbol_feed` (engine and codegen
39+
1.0.0 and later). One feed per requested timeframe: nothing aggregates another
40+
symbol's bars, so a `1` feed serves neither a `240` nor a `D` request. Unset
41+
changes nothing; an index or feed it cannot install fails
42+
the run (one `{"engine":"pineforge","error":...}` line, harness exit 1,
43+
entrypoint exit 4). What was installed is recorded as
44+
`applied_runtime.symbol_feeds`, so such a run has its own fingerprint digest.
45+
The engine library is unchanged.
46+
3447
- **Native runner routing and delivery:** webhooks are optional; `--webhook-routes`
3548
adds first-match per-action targets and payload `pineforge-native-order-action/v2`.
3649
New `actions`, `status` and offline `redeliver` commands expose the journal and

‎docker/README.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,88 @@ library's `strategy_set_syminfo_*` setters.
172172
`docker/run_json.py` is vendored: pineforge-release copies it from the engine tag
173173
at every release, so the lot-grid handling (`mincontract`) lives in this file.
174174

175+
### Other symbols' bars (`PINEFORGE_SYMBOL_FEEDS`)
176+
177+
A script that calls `request.security` on another symbol reads that symbol's
178+
own bars, never the chart's. Without them, or when the index below lacks the
179+
requested symbol string or timeframe, the run stops where the request's value
180+
is read (`request.security(...) at line N: no data is pinned for this request,
181+
and its value was read`, exit 4). `PINEFORGE_SYMBOL_FEEDS` (the
182+
harness's `--symbol-feeds`) names a JSON index of those bars, installed through
183+
the library's `strategy_set_symbol_facts` and `strategy_set_symbol_feed` (engine
184+
1.0.0 and later):
185+
186+
```json
187+
{"symbols": {
188+
"BINANCE:ETHUSDT": {
189+
"syminfo": {"tickerid": "BINANCE:ETHUSDT", "type": "crypto", "currency": "USDT",
190+
"mintick": 0.01, "session": "24x7", "timezone": "UTC"},
191+
"feeds": {"240": "ethusdt-240.csv", "1D": "ethusdt-1D.csv"}}}}
192+
```
193+
194+
```bash
195+
docker run --rm \
196+
-v $(pwd)/strategy.pine:/in/strategy.pine:ro \
197+
-v $(pwd)/btcusdt-240.csv:/in/ohlcv.csv:ro \
198+
-v $(pwd)/symbols:/in/symbols:ro \
199+
-e PINEFORGE_SYMBOL_FEEDS=/in/symbols/symbols.json \
200+
pineforge > report.json
201+
```
202+
203+
- A symbol key is the exact string the script passes at run time, exchange
204+
prefix and suffix included: `BINANCE:ETHUSDT`, `ETHUSDT` and
205+
`BINANCE:ETHUSDT.P` are three symbols. For `input.symbol` it is the input's
206+
value (its default, or the `PINEFORGE_INPUTS` override). A string naming the
207+
chart's own market is another symbol too: the harness does not set the chart's
208+
`syminfo.tickerid`.
209+
- One feed per timeframe the script requests, keyed in the engine's spelling:
210+
whole minutes as a bare integer (`"240"`, never `"4h"`), else `<n>D|W|M|S`;
211+
a bare `D`/`W`/`M`/`S` is folded to `1D`/`1W`/`1M`/`1S`. A request at
212+
`timeframe.period` (or `""`) reads the feed at the chart's timeframe.
213+
- A feed is a CSV like `ohlcv.csv` (`timestamp,open,high,low,close,volume`; an
214+
empty volume is a symbol that publishes none), paths relative to the index.
215+
Each bar's close is its open plus the timeframe (calendar months for `M`),
216+
right for a 24x7 symbol; give a session-bound symbol a `time_close` column
217+
(unix ms; an empty cell falls back to open plus timeframe). Bars before the
218+
chart's first bar are delivered as history on it; bars after its last are
219+
never read. A header-only feed installs the symbol without bars: its requests
220+
read na on every bar (a symbol with no bars in the window).
221+
- `syminfo` is the symbol's catalog object, flat or `{"syminfo": {...}}`. Its
222+
`type`, `timezone`, `session`, `currency` and `mintick` are set as the
223+
symbol's facts, which `syminfo.*` reads inside the request; `tickerid` is set
224+
as its `canonical` fact, which no `syminfo.*` reads; other keys are ignored.
225+
Inside the request `syminfo.tickerid` is always the key and `syminfo.ticker`
226+
the key after its last `:`. Without `syminfo`, `syminfo.mintick` reads NaN
227+
and those four strings read empty.
228+
- Merge rule (TradingView's): with `lookahead` off a chart bar reads the latest
229+
requested bar whose close is at or before the chart bar's close; with it on,
230+
the latest that opened at or before the chart bar's open. A missing requested
231+
bar carries the last value forward (`gaps` off) or reads na (`gaps` on).
232+
- Limits: at most 256 symbols and 256 feeds in one index. The chart must be its
233+
own input (`PINEFORGE_SCRIPT_TF` unset or equal to the input timeframe), else
234+
the run fails with `request.security of another symbol needs the chart's own
235+
bars as input; input '<i>' aggregated to chart '<s>' is not supported`. The run
236+
is historical only. Each feed is a full pass of its request's expression over
237+
its bars. `request.security_lower_tf` on another symbol reads no feed.
238+
- A feed must be at the timeframe it serves. Nothing aggregates another
239+
symbol's bars: the engine looks a feed up by the exact symbol string and
240+
timeframe, and `strategy_set_symbol_feed` installs the bars as given. A `240`
241+
feed does not serve a `D` request, and a `1` feed serves only a request at
242+
`1`, so give each requested timeframe its own bars.
243+
- An index or feed the harness cannot install (for example: not JSON, a bad
244+
timeframe spelling, two feeds at one timeframe, a CSV without `close`,
245+
timestamps that do not strictly increase, a close after the next bar's open,
246+
a non-positive `mintick`, more than 256 feeds, a library without the setters,
247+
a feed the engine refuses)
248+
fails the run before it starts: one line
249+
`{"engine":"pineforge","error":"--symbol-feeds: ..."}` on stdout, harness exit
250+
1, entrypoint exit 4.
251+
- What was installed is recorded as `applied_runtime.symbol_feeds` (each
252+
symbol's facts, and per feed its bar count, first and last open and a hash of
253+
its values), so the fingerprint digest differs from a run without it. Unset,
254+
or an index naming no symbol: the key is absent and the report is what it was
255+
before this variable existed, apart from `elapsed_seconds`.
256+
175257
The engine catches every error (TF mismatch, unsupported emulation
176258
flags, unknown-input-TF, etc.) into `strategy_get_last_error()`; the
177259
container surfaces these as `{"engine":"pineforge","error":"..."}` on

‎docker/entrypoint.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,14 @@
4343
# keys are ignored. mincontract absent or null: no lot grid. Any other
4444
# mincontract that is not a positive finite number fails the run
4545
# (exit 4, one {"engine":"pineforge","error":...} line on stdout).
46+
# PINEFORGE_SYMBOL_FEEDS Path to a JSON index of other symbols' bars for
47+
# request.security on another symbol: {"symbols": {"<symbol string>":
48+
# {"syminfo": {...}, "feeds": {"<timeframe>": "<csv path>"}}}}, the
49+
# symbol string exactly as the script passes it, one CSV per
50+
# timeframe the script requests (paths relative to the index). Unset:
51+
# nothing is installed and such a request stops the run where its
52+
# value is read. An index or feed the harness cannot install fails
53+
# the run (exit 4, one {"engine":"pineforge","error":...} line).
4654
#
4755
# Exit codes:
4856
# 0 success (JSON report, or C++ in transpile-only mode, on stdout)
@@ -140,12 +148,14 @@ echo "[pineforge] running backtest ..." >&2
140148
# PINEFORGE_CHART_TZ IANA tz for date builtins
141149
# PINEFORGE_MAGNIFIER_VOLUME_WEIGHTED 1/true → vw magnifier (needs BAR_MAGNIFIER)
142150
# PINEFORGE_SYMINFO path to a syminfo.json (see the header)
151+
# PINEFORGE_SYMBOL_FEEDS path to other symbols' feed index (see the header)
143152
# PINEFORGE_BENCH (+_WARMUP/_REPEATS) 1/true → timing mode
144153
extra=()
145154
[[ -n "${PINEFORGE_TRADE_START_MS:-}" ]] && extra+=(--trade-start-ms "${PINEFORGE_TRADE_START_MS}")
146155
[[ -n "${PINEFORGE_CHART_TZ:-}" ]] && extra+=(--chart-tz "${PINEFORGE_CHART_TZ}")
147156
[[ "${PINEFORGE_MAGNIFIER_VOLUME_WEIGHTED:-}" =~ ^(1|true|yes|on)$ ]] && extra+=(--magnifier-volume-weighted)
148157
[[ -n "${PINEFORGE_SYMINFO:-}" ]] && extra+=(--syminfo "${PINEFORGE_SYMINFO}")
158+
[[ -n "${PINEFORGE_SYMBOL_FEEDS:-}" ]] && extra+=(--symbol-feeds "${PINEFORGE_SYMBOL_FEEDS}")
149159
if [[ "${PINEFORGE_BENCH:-}" =~ ^(1|true|yes|on)$ ]]; then
150160
extra+=(--bench --warmup "${PINEFORGE_WARMUP:-3}" --repeats "${PINEFORGE_REPEATS:-20}")
151161
fi

0 commit comments

Comments
 (0)