Repository navigation
Commit 650b0cb
fix: report a position still open after the final bar as TradingView's range-end close
TradingView's deep-backtest report (the ws-report-v1 tape the campaign
grades against) does not leave the last position open. It reports it as
a closed trade whose exit leg is the range's last bar at that bar's
close, with an empty exit Signal, and counts it in closedTrades. The
canonical row is orb-lite on NYSE:F 1D: Entry short 2026-03-16 @ 11.82,
Exit 2026-04-30 @ 12.08 -- 12.08 is the last close of the range,
metrics closedTrades:1. On the f-1d spark set 8/10 engine runs held the
same position to the last bar (bars-in-market = TV Duration + 1) and
7/10 had a mark-to-market open_pl equal to TV's row to the cent: the
row is exactly the engine's open position marked at the last close.
The engine exported closed trades only (trades_ grows in
emit_close_trade alone; neither run loop flattened at end of feed), so
every such probe was one trade short against the tape and the verifier
scored the missing row as an unmatched TV trade. The browser export
writes the same row with Signal "Open", which the verifier already
recognises; the ws tape gives it no marker at all.
Operator decision (2026-09-02): the engine emulates TradingView's
range-end accounting, in the report. After the last script bar is
dispatched and its equity point recorded, record_range_end_close_trades
builds the rows a close of the open position at the last bar's close
would record -- one per pyramid slice through the same arithmetic as
every other close (build_close_trade, split out of emit_close_trade) --
into range_end_trades_, which fill_trades_section merges behind the
script's closed trades. The live position, pending orders, trades_ and
the realized sums are left as the bar loop left them: a stream's
realtime bars continue the warmup position, Pine accessors never see
the row, and the broker-mechanics tests keep inspecting the open lot.
Pricing: the row is a mark at the close, not an order the broker fills
-- the raw close rounded to the nearest tick (bar_fill_price,
finding-446; the tape's 12.08 is the on-tick close), no slippage ticks,
commission per the strategy's rules on the exit leg like any close (the
tape in hand has commission 0, so that is the operator's call, not a
measurement), dated on the script bar's label. Accounting: the report's
closed-trade count, net profit, win/loss tallies and commission paid
include the rows. Every equity point before the last is untouched; the
last point is re-marked to the flat account (open_profit 0, equity =
capital + net profit including the rows) so the report identity
equity == initial_capital + net_profit + open_pl that test_metrics pins
still holds on the last bar -- with the tape's commission 0 and an
on-tick close that point's equity VALUE is the gross mark it already
was. A first cut re-marked the point and left the scalar extremes at the
bar loop's gross fold; review showed that broke the compute_equity_stats
walk == scalar identity on every commissioned run whose last bar was the
trough or the peak (the fold had seen the gross mark, the walk the net
one), so the scalars are re-folded from the curve after the re-mark
(fold_equity_extreme, split out of update_equity_extremes): the walk and
the scalars agree again, and every earlier bar's drawdown / run-up is
unchanged. A strategy flat after the final bar is unaffected.
The measurement must END where TradingView's range ends, or the row is
booked on the wrong bar. The engine marks after the last bar it is FED,
and the ETH chart feed (eth-corpus-15 / eth-scraped-15) runs on to
2026-05-04 15:00 UTC while every ETH tape's range ends 2026-05-01 (228
of 397 scrapper tapes with an Open row at 05-01 00:00 UTC @ 2261.44 =
that bar's close); the ws-report lanes' feeds already end there.
run_strategy.py now bounds the feed (ohlcv_end_ms, on both runners; the
docker runner pre-trims through _trim_ohlcv_csv) to TradingView's RANGE
END whenever the TV window is in use and the tape is UNMARKED (the v4
rule, below): the bars opening at or before the
tape's `to` as 00:00 UTC, read from the metrics.json beside the tape
(wsProvenance.requestedRange.to when the ws exporter recorded the window
it sent, else the `to` date; _load_tv_range_end_ms). The rule is
TradingView's own, read off the ws tapes' returnedRange.to against the
requestedRange.to = 2026-05-01 00:00 UTC every campaign tape asked for:
BTC/ES/NQ/XAU 15 return 05-01 00:00 (the bar OPENING at `to` is inside,
00:15 is not -- on CME's America/Chicago chart too, so `to` is UTC
midnight, not the chart tz's), NYSE:F 1D returns 04-30 13:30 (the 05-01
13:30 session bar exists and is excluded), ES/NQ 1D 04-30 22:00, XAU 1D
04-30 21:00, NIFTY 15/1D 04-30 09:45/03:45, F 15 04-30 19:45 -- and every
lane's open-position row sits on exactly that bar (survey of the
scrapper's 17 lane directories, 1,356 tapes, 2026-09-02).
Not the tape's last trade row. A first cut bounded the feed there, and
the full-population gate (gate-cand-range-end-close-20260902) failed
hard.regression on scrapper:data/standard/waranyutrkm-asian-box-breakout-
eda-tuned (ETH 15): 110 -> 109 trades, match 100 -> 99.1. Its last trade
closes 04-29 18:00 UTC, so the feed was cut mid-day, and the strategy
gates entries on request.security(.., "D", close / ta.ema(close, 50),
lookahead=barmerge.lookahead_on): under the historical lookahead
projection (the probe's winning verifier candidate) every 04-29 chart
bar reads the 04-29 daily bar's FINAL value. Cut at 18:00 the partial day
gave d_close 2240.35 < d_ema 2241.85 and no long; the complete day, as TV
sees it with its range running to 05-01, gives 2251.52 > 2242.29 and
TV's trade 110 at 09:45 (spark, Route A: bound at the tape row 109 trades
sha dc086072, bound at 04-29 23:45 / 05-01 00:00 / unbounded 110 trades
sha c5ebd23f = the Cloud Run baseline; the four other near-range-end
closed probes in the shard are byte-identical either way). Any probe
with a lookahead_on HTF projection and a closed last trade inside the
final HTF period is exposed the same way. A metrics.json without a range
falls back to the tape's latest row. Trades ENTERED after TV's last
entry are still dropped by _filter_trades_to_window.
The auxiliary request.security feed (the ETH lane's 1m FEED_1M) is left
as exported: the engine's routing ignores aux bars labelled past the
last chart bar (engine_aux_security.cpp tail prefilter), so the chart's
last bar keeps its full slice and no chart bar is lost;
test_aux_security_feed pins that on the 15m chart / 1m aux shape.
The row reaches engine_trades.csv only when the tape prints it the same
way, and WHERE the measurement ends is the tape's call too. Two exporters
write the campaign's tapes: ws-report-v1 (every 1D lane and most 15m;
286 of the 305 ETH tapes in the ws lanes; no marker in metrics.json
beyond tapeChannel) reports the range-end position as an ordinary closed
trade with an empty Signal, the shape the emulation targets; the browser
export (metrics.json without tapeChannel; 227 of the 396 scraped ETH
tapes) writes the same row with Signal "Open". TV nets commission on
BOTH legs of an Open row exactly as on a close -- 3commas-xlm-grid-bot
#615: qty 0.0912, entry 2189.13, mark 2261.44 -> 6.3511 = TV's 6.35 =
the engine's 6.351137 -- so per-lot P&L is exact to the cent; the
difference is what the grader does with the marker.
The v4 rule (2026-09-02), keyed on the marker (_tv_tape_marks_open_rows:
any row whose Signal.strip().lower() is "open"; _apply_range_end_regime
applies it and main() logs one line saying which regime applied and
why):
UNMARKED tape -> the range-end regime (v2): bound the feed at TV's
range end as above and write the range-end close rows. Every unmarked
lane measured was clean under it: f-1d 10/10 engine-0 -> excellent,
waranyutrkm asian-box 110/110 at 100%, and the 8 unmarked range-end
ETH tapes byte-identical (spark smoke-m2v3 / smoke-m2v3b / f-1d).
MARKED tape -> the BASELINE behaviour: no ohlcv_end_ms bound (the feed
as exported, three days past every ETH tape's range) and no
open_at_end row written -- the rows are withheld from engine_trades.csv
(_withhold_open_at_end_trades; with an unbounded feed a position may
still be open at the FEED's end, and that mark is withheld like any
other), logging the count withheld and why and counting it on the
summary line. The CSV is then the baseline's to the byte.
The marked regime is the baseline's because the baseline traded past
TV's range end and the canonical grader (scripts/verify_corpus.py)
pairs those post-range closes with TV's Open rows on ENTRY time -- the
match the campaign's scores are built on. Two bounded variants were
measured on spark against it. v2 (bound + emit on every tape) paired
the marks exactly but moved pnlP90 on the three 3commas grid bots by
the cent-rounding of TV's Net PnL -- gate v2 4gfpn on tree ad21d0e:
hard lane 5 better / 696 same / 3 worse, scrapper:data/standard/
3commas-{3commas-xlm,btcusdt-p,nvdax}-grid-bot-long-strategy matchPct
94.3 -> 100 but pnlP90 0.55 -> 0.67, 0.53 -> 0.64, 0.18 -> 0.93 (22/22/28
open lots each, marked Open at 2026-05-01 08:00 +8) -- and the hard gate
counts any metric increase as a regression. v3 (bound + withhold on
marked tapes) turned every post-range close into a withheld open_at_end
row and lost the pairing on ~186 marked probes (3commas-xlm 94.3 -> 93.7
strong, alexgrover 100 -> 99.8; spark smoke-m2v3 pre-check). Until
verify_corpus.py pairs the marks itself (the follow-up), a marked tape
is measured exactly as the baseline measured it. The verifier grades
the CSV alone (run_strategy.py writes no report JSON), so the CSV is
the whole of what it sees. The range-end engine mechanics are the same
under both regimes.
The browser export's Open row can also be priced at the export-time
quote rather than a bar's print: all 12 sampled OANDA:EURUSD tapes end
with Open rows @ 1.17256 while the feed's last bar is o 1.17289 h 1.17299
l 1.17282 c 1.1729 (246 of 358 EURUSD tapes end with such a row).
verify_corpus.py treats a TV Open row whose engine counterpart exits on
the SAME bar as count-only (counted, covered, entry compared; exit/pnl
deltas left out and the pair reported as open_mark_pairs); an engine exit
on a different bar keeps its deltas, since TV held to the end and the
engine did not. With the harness withholding the row against a marked
tape that pair no longer forms on a normal run; the rule stays for a CSV
written without a tape window (--no-trim-output / --emit-window-ohlcv),
where the row is still emitted. pineforge-utils/validator/validate.py
tracks this rubric and needs the same rule.
pf_trade_t gains open_at_end (1 on the range-end row) -- ABI v3, with
the ctypes mirrors (run_strategy, docker/run_json, tutorial, benchmarks)
and the docs updated; run_strategy.py carries the flag on the report
dict and writes the CSV row as an ordinary exit with no marker, the way
the ws tape prints it, or withholds it against a browser tape as above.
run_stream_corpus compares the trades the two brokers made, without the
range-end rows.
tests/test_range_end_close.cpp pins the row (long, short, flat, pyramid
slices, commission on both legs, rounded close without slippage,
aggregated-path label, earlier curve points untouched and the last
re-marked flat, walk == scalar extremes
with commission and the last bar as trough and as peak, re-run reset);
test_run_inputs_overrides and test_accounting_reconciliation foot the
report to the blotter plus the range-end rows;
scripts/test_run_strategy_range_end.py pins the range-end rule (the ws
integer and the browser date, every lane's returnedRange under it), the
bound on the asian-box shape (a closed last trade before the range end
keeps the whole final day and every bar through TV's last bar, where the
tape-row bound cut it at 18:00), the Open-row shape (the last bar the
engine sees is the bar and price of TV's Open row), the NYSE:F 1D shape
(the 04-30 bar kept, a 05-01 session bar dropped), the fallback to the
tape's last row without a range, and the harness's EXPECTED_PF_ABI to
the header's PF_ABI_VERSION (the f-1d spark pre-check of 2026-09-02 failed all 64
probes with ".so reports 3, harness expects 2" while that guard was
still 2 -- the verifier runs every probe through run_strategy.py), and
the v4 regime through _apply_range_end_regime and through main() with
the engine stubbed: a marked tape sets no bound, withholds the
open_at_end rows (the mark at the feed's end included) and leaves the
closed rows -- a post-range close among them -- intact with the count
logged; an unmarked tape is bounded at the range end (the ws integer,
the browser date, the tape's last row as the fallback, a probe's own
ohlcv_end_ms kept) and writes the rows; a header-only tape sets no
bound and writes every row; and the one log line names the regime and
its source; scripts/test_verify_corpus_open_mark.py pins the EURUSD
Open-row pairing.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01BXAE7TVTKZYbmtenHDHVJo1 parent bc78cc7 commit 650b0cb
24 files changed
Lines changed: 2207 additions & 62 deletions
File tree
- benchmarks/throughput
- docker
- docs/pages
- include/pineforge
- scripts
- src
- tests
- tutorial
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
354 | 354 | | |
355 | 355 | | |
356 | 356 | | |
357 | | - | |
| 357 | + | |
358 | 358 | | |
359 | 359 | | |
360 | 360 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
127 | 128 | | |
128 | 129 | | |
129 | 130 | | |
130 | | - | |
| 131 | + | |
131 | 132 | | |
132 | 133 | | |
133 | 134 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
| |||
575 | 577 | | |
576 | 578 | | |
577 | 579 | | |
| 580 | + | |
578 | 581 | | |
579 | 582 | | |
580 | 583 | | |
| |||
701 | 704 | | |
702 | 705 | | |
703 | 706 | | |
704 | | - | |
| 707 | + | |
705 | 708 | | |
706 | 709 | | |
707 | 710 | | |
| |||
869 | 872 | | |
870 | 873 | | |
871 | 874 | | |
| 875 | + | |
872 | 876 | | |
873 | 877 | | |
874 | 878 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
63 | 66 | | |
64 | 67 | | |
65 | 68 | | |
| |||
165 | 168 | | |
166 | 169 | | |
167 | 170 | | |
168 | | - | |
| 171 | + | |
169 | 172 | | |
170 | 173 | | |
171 | 174 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
| |||
177 | 178 | | |
178 | 179 | | |
179 | 180 | | |
180 | | - | |
| 181 | + | |
181 | 182 | | |
182 | 183 | | |
183 | 184 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
| 61 | + | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
63 | 65 | | |
64 | 66 | | |
65 | 67 | | |
| |||
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
91 | 96 | | |
92 | 97 | | |
93 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
108 | 112 | | |
109 | 113 | | |
110 | 114 | | |
| |||
124 | 128 | | |
125 | 129 | | |
126 | 130 | | |
| 131 | + | |
127 | 132 | | |
128 | 133 | | |
129 | 134 | | |
| |||
1125 | 1130 | | |
1126 | 1131 | | |
1127 | 1132 | | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
1128 | 1139 | | |
1129 | 1140 | | |
1130 | 1141 | | |
| |||
2388 | 2399 | | |
2389 | 2400 | | |
2390 | 2401 | | |
2391 | | - | |
2392 | | - | |
2393 | | - | |
| 2402 | + | |
| 2403 | + | |
| 2404 | + | |
| 2405 | + | |
| 2406 | + | |
| 2407 | + | |
| 2408 | + | |
2394 | 2409 | | |
2395 | 2410 | | |
2396 | 2411 | | |
| |||
2402 | 2417 | | |
2403 | 2418 | | |
2404 | 2419 | | |
| 2420 | + | |
| 2421 | + | |
| 2422 | + | |
2405 | 2423 | | |
2406 | 2424 | | |
2407 | 2425 | | |
| |||
2578 | 2596 | | |
2579 | 2597 | | |
2580 | 2598 | | |
| 2599 | + | |
| 2600 | + | |
| 2601 | + | |
| 2602 | + | |
| 2603 | + | |
| 2604 | + | |
| 2605 | + | |
| 2606 | + | |
2581 | 2607 | | |
2582 | 2608 | | |
2583 | 2609 | | |
| |||
2818 | 2844 | | |
2819 | 2845 | | |
2820 | 2846 | | |
| 2847 | + | |
| 2848 | + | |
| 2849 | + | |
| 2850 | + | |
| 2851 | + | |
| 2852 | + | |
| 2853 | + | |
2821 | 2854 | | |
2822 | 2855 | | |
2823 | 2856 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
78 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
| |||
156 | 158 | | |
157 | 159 | | |
158 | 160 | | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
159 | 176 | | |
160 | 177 | | |
161 | 178 | | |
| |||
314 | 331 | | |
315 | 332 | | |
316 | 333 | | |
317 | | - | |
318 | | - | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
319 | 339 | | |
320 | 340 | | |
321 | 341 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
155 | 155 | | |
156 | 156 | | |
157 | 157 | | |
| 158 | + | |
158 | 159 | | |
159 | 160 | | |
160 | 161 | | |
0 commit comments