|
| 1 | +# PineForge Engine — Runtime Overrides & Order-Execution Cheatsheet |
| 2 | + |
| 3 | +> **Source-verified.** Every claim was adversarially checked against the code |
| 4 | +> (file:line) on `feat/input-source-syminfo-issues` (engine) / |
| 5 | +> `feat/input-source-color-syminfo` (codegen). Corrections from that pass are |
| 6 | +> folded in (notably: `engine_chart_timezone` is validator-only not a harness |
| 7 | +> key; the `pending_close_qty_in_bar_` reset is in the outer bar loop not |
| 8 | +> `dispatch_bar`; stop-entry mintick rounding is skipped on gap fills; |
| 9 | +> `PERCENT_OF_EQUITY` includes open profit only on the explicit-`qty_type` path). |
| 10 | +
|
| 11 | +Reference for driving the engine and for building a **forward / real-time** |
| 12 | +executor that matches backtest semantics. Two parts: |
| 13 | + |
| 14 | +1. [Runtime-overrideable parameters](#part-1--runtime-overrideable-parameters) |
| 15 | +2. [Order-execution rules](#part-2--order-execution-rules) (the hard part) |
| 16 | + |
| 17 | +> Surface note: `get_input_source`, `strategy_set_syminfo_timezone/_session/_metadata` |
| 18 | +> land with the input.source/syminfo feature (engine PR #25 + codegen PR #18). |
| 19 | +> Everything else is on `main`. File:line refs are approximate — grep the symbol. |
| 20 | +
|
| 21 | +--- |
| 22 | + |
| 23 | +# Part 1 — Runtime-overrideable parameters |
| 24 | + |
| 25 | +"Runtime" = settable on a strategy handle **before `run_backtest*`**, without |
| 26 | +recompiling. "Compile-time" = baked into `generated.cpp` from the Pine |
| 27 | +`strategy(...)` / `strategy.risk.*` source; only changeable by re-transpiling. |
| 28 | + |
| 29 | +## 1.1 C-ABI setters (call before run; persist across runs on the same handle) |
| 30 | + |
| 31 | +| Setter | Default | Semantics | |
| 32 | +|---|---|---| |
| 33 | +| `strategy_set_input(s, key, value)` | — | Override a Pine `input.*()` by **title** (key = input title string). Value serialized as string. | |
| 34 | +| `strategy_set_override(s, key, value)` | — | Override a `strategy()` decl param (see 1.2 for the 9 keys). | |
| 35 | +| `strategy_set_chart_timezone(s, tz)` | `""`=UTC | Chart **display** TZ (IANA). Drives `hour/minute/dayofweek/...` builtins **and intraday-cap day rollover**. `""`/`UTC`/`Etc/UTC` = fast `gmtime_r`; else process-global mutex. | |
| 36 | +| `strategy_set_trade_start_time(s, ms)` | `INT64_MIN` (no gate) | Suppress `entry/order/exit/close` before this Unix-ms; TA still runs. | |
| 37 | +| `strategy_set_trace_enabled(s, on)` | `0` | Capture `// @pf-trace` records into the report. | |
| 38 | +| `strategy_set_magnifier_volume_weighted(s, on)` | `0` | Volume-weighted sub-bar sampling (only if magnifier on). | |
| 39 | +| `strategy_set_syminfo_timezone(s, tz)` | `UTC` | Exchange TZ → feeds `session.ismarket` / `time(session)`. **Separate slot** from chart tz. | |
| 40 | +| `strategy_set_syminfo_session(s, str)` | `24x7` | Session string e.g. `0930-1600:23456` (days 1=Sun..7=Sat). | |
| 41 | +| `strategy_set_syminfo_metadata(s, key, double)` | reads `na` | Inject a fundamental field (`shares_outstanding_total`, `target_price_*`, …); engine only computes — data fed externally. | |
| 42 | + |
| 43 | +Run entry points: `run_backtest(s,bars,n,out)` (auto-detect TF, no magnifier) / |
| 44 | +`run_backtest_full(s,bars,n,input_tf,script_tf,bar_magnifier,magnifier_samples,magnifier_dist,out)`. |
| 45 | +Always check `strategy_get_last_error(s)` after. `report_free(&out)` to free. |
| 46 | + |
| 47 | +**Persistence:** `inputs_`, `chart_timezone_`, `trade_start_time_`, |
| 48 | +`trace_enabled_`, overrides, syminfo all persist across runs on the same handle. |
| 49 | +**Risk state is NOT auto-reset** between runs on a reused handle. |
| 50 | + |
| 51 | +## 1.2 `strategy()` params — runtime-overrideable via `strategy_set_override` |
| 52 | + |
| 53 | +Baked from the `.pine` at compile-time; these 9 can be overridden at runtime. |
| 54 | +Sentinels for "not overridden": `NaN` (doubles), `-1` (ints). |
| 55 | + |
| 56 | +| Key | Default | Values | |
| 57 | +|---|---|---| |
| 58 | +| `initial_capital` | 1,000,000 | decimal | |
| 59 | +| `commission_value` | 0 | decimal | |
| 60 | +| `commission_type` | percent(0) | `percent`/`0`, `cash_per_order`/`1`, `cash_per_contract`/`2` | |
| 61 | +| `default_qty_value` | 1 | decimal | |
| 62 | +| `default_qty_type` | fixed(0) | `fixed`/`0`, `percent_of_equity`/`1`, `cash`/`2` | |
| 63 | +| `pyramiding` | 1 | int | |
| 64 | +| `slippage` | 0 | int (ticks) | |
| 65 | +| `process_orders_on_close` | false | `true`/`1`/`false`/`0` | |
| 66 | +| `close_entries_rule` | FIFO | `ANY`/`any`/`1` = ANY, else FIFO | |
| 67 | + |
| 68 | +**Compile-time only (no override path):** `margin_long` / `margin_short` |
| 69 | +(default 100 = 1x), `script_has_strategy_close_` (gates the deferred-flip rule). |
| 70 | + |
| 71 | +## 1.3 `strategy.risk.*` — compile-time only |
| 72 | + |
| 73 | +Set inside `on_bar` by generated code; no ABI setter. All default to |
| 74 | +unlimited/both: |
| 75 | +`max_intraday_filled_orders`, `max_intraday_loss` (+is_pct), `max_drawdown` |
| 76 | +(+is_pct), `max_position_size`, `allow_entry_in` (direction lock), |
| 77 | +`max_cons_loss_days`. Gate: `check_risk_allow_entry`; halt latch: |
| 78 | +`update_risk_state`. |
| 79 | + |
| 80 | +## 1.4 Timeframe / magnifier (via `run_backtest_full`) |
| 81 | + |
| 82 | +| Param | Default | Notes | |
| 83 | +|---|---|---| |
| 84 | +| `input_tf` | `""` auto-detect | feed TF (`1`,`5`,`15`,`60`,`1D`…) | |
| 85 | +| `script_tf` | = input_tf | chart TF; must be ≥ input_tf (aggregates up) | |
| 86 | +| `bar_magnifier` | off | enable intrabar sub-bar walk | |
| 87 | +| `magnifier_samples` | 4 | ticks/parent bar (synthetic mode) | |
| 88 | +| `magnifier_dist` | ENDPOINTS(3) | 0 UNIFORM,1 COSINE,2 TRIANGLE,3 ENDPOINTS,4 FRONT_LOADED,5 BACK_LOADED. With **real** sub-bars, forced to ENDPOINTS + 4 samples. | |
| 89 | + |
| 90 | +## 1.5 SymInfo fields |
| 91 | + |
| 92 | +`ticker, tickerid, currency, basecurrency, type, timezone(UTC), session(24x7), |
| 93 | +volumetype(base), description, mintick(0.01), pointvalue(1.0)`. `mintick` drives |
| 94 | +slippage + directional rounding. Set via the `SymInfo`-taking `run()` overload, |
| 95 | +or `timezone`/`session`/metadata via the C-ABI setters in 1.1. |
| 96 | + |
| 97 | +## 1.6 Harness (`scripts/run_strategy.py`) — `inputs.json` keys |
| 98 | + |
| 99 | +Any non-meta key → `strategy_set_input`. Recognized: `ohlcv_csv`, |
| 100 | +`ohlcv_start_ms`, `script_tf`, `input_tf`, `chart_timezone`, and a |
| 101 | +`runtime_overrides` block: `bar_magnifier`, `magnifier_distribution`, |
| 102 | +`magnifier_samples`, `magnifier_volume_weighted`, `timezone`, `session`, |
| 103 | +`syminfo_metadata`. `tv_trades_csv` / `tv_trades_csv_tz` are verify-only. |
| 104 | + |
| 105 | +> `engine_chart_timezone` is **NOT** a `run_strategy.py` key — it's consumed |
| 106 | +> only by the private canonical validator. In this harness use `chart_timezone`. |
| 107 | +> (Verified: `engine_chart_timezone` appears nowhere in `scripts/run_strategy.py`.) |
| 108 | +
|
| 109 | +--- |
| 110 | + |
| 111 | +# Part 2 — Order-execution rules |
| 112 | + |
| 113 | +For a forward executor: the engine is a per-bar state machine. Below is the |
| 114 | +exact sequence, fill math, and the subtleties that bite on real-time replay. |
| 115 | + |
| 116 | +## 2.1 Per-bar order of operations |
| 117 | + |
| 118 | +`dispatch_bar()`: |
| 119 | + |
| 120 | +**`process_orders_on_close = false` (default):** |
| 121 | +> Step 1 runs in the **outer bar loop** (`engine_run.cpp:96 / 367 / 443`), |
| 122 | +> *before* `dispatch_bar()` — `dispatch_bar()` itself does not reset it. |
| 123 | +``` |
| 124 | +1. pending_close_qty_in_bar_ = 0 # reset (outer loop, pre-dispatch_bar) |
| 125 | +2. process_pending_orders(bar) # OLD stop/limit/market from prior bars |
| 126 | + a. update_risk_state() # may latch risk_halted_ |
| 127 | + b. update_trail_best_for_bar_open(bar) |
| 128 | + c. sort_exit_siblings_by_path_fill(bar) |
| 129 | + d. sort_orders_by_fill_phase(bar) # gap(0) before intrabar(1) |
| 130 | + e. dual-pass opposing-stop loop (pass 0 / pass 1) |
| 131 | + f. purge exit orders if now FLAT |
| 132 | +3. update_per_trade_extremes() # MFE/MAE current before strategy reads |
| 133 | +4. on_bar(bar) # strategy logic; creates NEW orders |
| 134 | +5. update_equity_extremes() |
| 135 | + # NEW market orders wait for NEXT bar's open |
| 136 | +``` |
| 137 | + |
| 138 | +**`process_orders_on_close = true`:** identical 1–4, then a **step 4b |
| 139 | +`process_pending_orders(bar)`** so NEW market orders fill at **this bar's |
| 140 | +close**. New *priced* (stop/limit/trail) orders created this bar are always |
| 141 | +skipped from that second pass (they wait for next bar). |
| 142 | + |
| 143 | +**Magnifier:** per sub-bar, per sample tick → `process_pending_orders` + |
| 144 | +`update_per_trade_extremes`; `on_bar` runs **once**, on the last tick of the |
| 145 | +last sub-bar (`is_first_tick_` forced true there). |
| 146 | + |
| 147 | +## 2.2 Fill price by order type |
| 148 | + |
| 149 | +All comparisons **non-strict** (`<=`/`>=`). After slippage, fills snap to |
| 150 | +mintick directionally (buys ceil, sells floor). |
| 151 | + |
| 152 | +| Order | Trigger | Fill price | Gap-at-open | |
| 153 | +|---|---|---|---| |
| 154 | +| Market | always | `close` (POOC on) / `open` (POOC off) | — | |
| 155 | +| Long limit entry | `low ≤ L` | `min(open, L)` | open if `open ≤ L` | |
| 156 | +| Short limit entry | `high ≥ L` | `max(open, L)` | open if `open ≥ L` | |
| 157 | +| Long stop entry | `high ≥ S` | `max(open, S)`, then ceil-mintick **if fill>open** (skipped on gap) | open if `open ≥ S` | |
| 158 | +| Short stop entry | `low ≤ S` | `min(open, S)`, then floor-mintick **if fill<open** (skipped on gap) | open if `open ≤ S` | |
| 159 | +| Stop-limit entry | path crosses S then L | activation pt or L | complex (§2.4) | |
| 160 | +| Long stop exit | `low ≤ S` | `S` (or open if gap) | open if `open ≤ S` | |
| 161 | +| Long limit exit | `high ≥ L` | `L` (or open if gap) | open if `open ≥ L` | |
| 162 | +| Short stop exit | `high ≥ S` | `S` (or open if gap) | open if `open ≥ S` | |
| 163 | +| Short limit exit | `low ≤ L` | `L` (or open if gap) | open if `open ≤ L` | |
| 164 | +| Trail exit (long) | path descends through `best − offset` | trail level (or open if gap) | open if `open ≤ trail` | |
| 165 | + |
| 166 | +**Gap shortcut suppressed on the entry bar** unless the magnifier is active |
| 167 | +(then every sub-bar open is a fresh gap event). |
| 168 | + |
| 169 | +## 2.3 Market-order timing (critical) |
| 170 | + |
| 171 | +- POOC **off**: market from `on_bar` → `pending_orders_`, fills **next bar |
| 172 | + open**. |
| 173 | +- POOC **on**: market from `on_bar` (no stop/limit) → `execute_market_entry` |
| 174 | + **immediately at this bar's close** (never queued). `strategy.close` likewise |
| 175 | + closes immediately at close. |
| 176 | +- Priced order from `on_bar`: queued, evaluated from **next bar** at step 2. |
| 177 | + |
| 178 | +## 2.4 Intrabar price path |
| 179 | + |
| 180 | +- Direction: `|open−high| < |open−low|` ⇒ path `O→H→L→C`, else `O→L→H→C` |
| 181 | + (proximity, not candle color). |
| 182 | +- 4 waypoints; same-segment priority **STOP > TRAIL > LIMIT**; within a segment, |
| 183 | + parametric position `t` decides; tie → STOP first. |
| 184 | +- Long position: stop active on **falling** segments, limit on **rising** (short |
| 185 | + reversed). |
| 186 | +- Two opposing entry stops from flat: `dual_entry_stop_path_winner` picks who |
| 187 | + fires first; loser deferred to pass 1; tie → long first. On pass 1, a trailing |
| 188 | + short-stop fills as a bracket close; a trailing long-stop after `ShortFirst` is |
| 189 | + discarded (TV only treats a same-bar second touch as a bracket exit when the |
| 190 | + buy-stop leads). |
| 191 | + |
| 192 | +## 2.5 Slippage & commission |
| 193 | + |
| 194 | +- Slippage = `slippage_ × mintick`; buy +slip then ceil-mintick, sell −slip then |
| 195 | + floor-mintick. `slippage_=0` still applies directional mintick rounding. |
| 196 | +- Commission per fill: `percent` = `price·qty·v/100`; `cash_per_order` = `v`; |
| 197 | + `cash_per_contract` = `v·qty`. **Both entry and exit** commissions deducted in |
| 198 | + `emit_close_trade`. |
| 199 | +- Raw PnL: long `(exit−entry)·qty`, short `(entry−exit)·qty`. |
| 200 | + |
| 201 | +## 2.6 Entry / position state machine |
| 202 | + |
| 203 | +`execute_market_entry` dispatch: |
| 204 | +- FLAT → `enter_market_from_flat` |
| 205 | +- same side → `add_to_pyramid_market` |
| 206 | +- flat-issued entry while opposite open (`close_only_opposite`) → |
| 207 | + `close_opposite_then_enter` (TV: flat-issued bracket entry **closes**, not |
| 208 | + reverses) |
| 209 | +- opposite → `flip_market_position_to` (new_size = explicit qty) |
| 210 | + |
| 211 | +**Deferred-flip carry (`tv_carry_qty`, `|old|+qty`):** a **priced** entry firing |
| 212 | +**from flat** opposite to its `created_position_side` opens at `carry + base_qty` |
| 213 | +**iff** all hold: `script_has_strategy_close_`, priced entry, `carry > 0`, |
| 214 | +opposite direction. Carry snapshot taken **at `strategy.entry` call time** = |
| 215 | +`max(0, position_qty_ − pending_close_qty_in_bar_)`. Same-id replacement |
| 216 | +re-snaps carry; siblings in the same cycle get carry zeroed on fire. |
| 217 | + |
| 218 | +**`strategy.entry` vs `strategy.order`:** entry respects pyramiding + margin |
| 219 | +gate + deferred-flip; order is `RAW_ORDER`, no margin check, no carry |
| 220 | +subtraction. |
| 221 | + |
| 222 | +## 2.7 Pyramiding |
| 223 | + |
| 224 | +Fill blocked if `position_entry_count_ ≥ pyramiding_` **except**: |
| 225 | +`flat_armed_priced` (placed from flat, now same dir) and |
| 226 | +`pre_armed_opposite_priced` (placed while opposite). One priced ENTRY per bar |
| 227 | +(`priced_entry_filled_this_bar_`) unless flat-armed/opposite-sibling. |
| 228 | + |
| 229 | +## 2.8 Sizing |
| 230 | + |
| 231 | +`FIXED`=value; `PERCENT_OF_EQUITY`=`equity·v/100 / fill`; `CASH`=`v / fill`. |
| 232 | +**Two equity definitions** (verified): explicit `qty_type=percent_of_equity` via |
| 233 | +`strategy.entry` → `calc_qty_for_type`, `equity = initial_capital_ + |
| 234 | +net_profit_sum_ + open_profit(close)` (**includes open profit**); the default |
| 235 | +`calc_qty` path → `equity = initial_capital_ + net_profit_sum_` (closed PnL |
| 236 | +only). `max_position_size` blocks at fill when `position_qty_ ≥ cap`. |
| 237 | + |
| 238 | +## 2.9 Exits, brackets, OCA |
| 239 | + |
| 240 | +- `strategy.exit` → `OrderType::EXIT` bound to `from_entry`; same-id replaces |
| 241 | + (keeps queue slot); partial exits one-shot per live position per id. |
| 242 | +- `strategy.close[_all]`: immediate at close if POOC/immediately, else deferred |
| 243 | + `__close__<id>` filling next open; bumps `pending_close_qty_in_bar_`; a full |
| 244 | + close cancels prior-bar adds to the closing position (keeps same-bar reversal |
| 245 | + entries). |
| 246 | +- FIFO (default) drains `pyramid_entries_` front-to-back; ANY closes only |
| 247 | + matching `entry_id`. |
| 248 | +- Same-bar multi-level: exit siblings path-sorted (earliest touch; full before |
| 249 | + partial on trail; tie = decl order) **before** the phase sort. |
| 250 | +- OCA cancel(1): full-fill of a sibling erases the group. OCA reduce(2): sibling |
| 251 | + qty reduced by filled; `strategy.exit` with oca → type 1. |
| 252 | + |
| 253 | +## 2.10 Same-bar close-then-entry |
| 254 | + |
| 255 | +`pending_close_qty_in_bar_` accumulates `strategy.close*` qty during `on_bar`; |
| 256 | +a later `strategy.entry` sees post-close carry. Full-close market exit sorts |
| 257 | +**before** opposite-direction priced entry so the engine is flat when the |
| 258 | +deferred-flip entry fires. Reset to 0 at bar start. |
| 259 | + |
| 260 | +## 2.11 Risk gates — when & what |
| 261 | + |
| 262 | +- `update_risk_state()` at **start of `process_pending_orders`** (not |
| 263 | + placement): drawdown halt, intraday-loss halt, cons-loss-day halt. Once |
| 264 | + `risk_halted_`, blocks all entries. |
| 265 | +- `check_risk_allow_entry` at fill (and in `execute_market_entry`): halt, |
| 266 | + direction lock, max_position_size. |
| 267 | +- Margin gate at **placement** (market entry w/ explicit qty only): |
| 268 | + `|qty|·signal_close·margin%/100 ≤ equity(closed)` else silently dropped. |
| 269 | +- `max_intraday_filled_orders`: counted at fill, keyed `dayofmonth*100+month` in |
| 270 | + **chart tz**; Nth fill emits a synthetic "Close Position (max filled orders)" |
| 271 | + exit then **latches** until day rollover; while latched, fills **and |
| 272 | + placements** dropped. Cap-close price = `bar.high/low` if the trigger was an |
| 273 | + intrabar stop, else `fill_price`. |
| 274 | + |
| 275 | +## 2.12 Trade record & stamping |
| 276 | + |
| 277 | +`emit_close_trade` (single close convergence): `entry_time = pe.time`, |
| 278 | +`exit_time = current_bar_.timestamp`, slippage-adjusted prices, PnL net of both |
| 279 | +commissions, `entry_bar_index`/`exit_bar_index`, MFE/MAE from the pyramid entry. |
| 280 | +`net_profit_sum_`, win/loss/even counts, intraday PnL, cons-loss-day all updated |
| 281 | +here. exit_id/comment stamped by the caller post-fill. |
| 282 | + |
| 283 | +## 2.13 Time/index stamping — magnifier caveat |
| 284 | + |
| 285 | +`bar_index_` = script-bar index (fixed across the whole magnifier loop). |
| 286 | +`current_bar_.timestamp` under the magnifier = the **sub-bar** timestamp → a |
| 287 | +fill mid-sub-bar stamps `entry_time`/`exit_time` with that sub-bar's time, not |
| 288 | +the script-bar open. (Aggregation-path label uses first-present sub-bar |
| 289 | +timestamp — see `engine_run.cpp` dispatch comment.) |
| 290 | + |
| 291 | +--- |
| 292 | + |
| 293 | +## Forward / real-time replay — gotchas checklist |
| 294 | + |
| 295 | +1. Reset `pending_close_qty_in_bar_ = 0` **before** running strategy logic each bar. |
| 296 | +2. Call `update_risk_state()` before the fill loop, not at placement. |
| 297 | +3. Keep the **two** sequential sorts (exit-sibling path sort, then phase sort). |
| 298 | +4. Snapshot `tv_carry_qty` at command-issue time, not fill time. |
| 299 | +5. `trail_best_price_` updates at bar open, again after each entry fill. |
| 300 | +6. On a flip, un-slip raw price then re-slip per new direction (exit uses old dir, entry new dir). |
| 301 | +7. Gap-at-open shortcut is OFF on the entry bar unless magnifier active. |
| 302 | +8. OCA-cancel only after a sibling **fully** fills (partial TP keeps SL alive). |
| 303 | +9. Intraday-cap latch resets only on **chart-tz** day rollover. |
| 304 | +10. Under LTF/real-time, stamp fills with the **sub-bar** time, not the aggregate bar. |
| 305 | +11. Market vs priced timing: market next-open (POOC off) / this-close (POOC on); priced waits a bar. |
| 306 | +12. Reused handle keeps risk state — reset/recreate per session if you don't want carryover. |
0 commit comments