Skip to content

codegen: a calc_on_order_fills restore no longer rolls back requested-evaluation state - #260

Closed
luisleo526 wants to merge 4 commits into
mainfrom
fix/calc-on-fills-late-restore
Closed

luisleo526 wants to merge 4 commits into
mainfrom
fix/calc-on-fills-late-restore

Conversation

@luisleo526

Copy link
Copy Markdown
Contributor

What

Under calc_on_order_fills a fill recalculation that runs after a deferred request.security evaluation (for example a late margin-call liquidation) restored the strategy checkpoint taken before that evaluation. The generated state of the request context (helper-local var series and their seeds, requested TA objects, OHLC and expression histories, mutable copies, bar-index clocks) lost the deferred writes while the engine's requested aggregators had already advanced: a silent wrong value (a helper counter read 11 where the minute feed gives 13).

The generated restore_script_state() now skips exactly those members, taken from the same metadata that declares them, and still restores chart state and the merged request results. Snapshot and commit are unchanged. No engine change, no ABI change.

Scope and limits

  • A site whose payload keeps user calls on the chart's bar (the chart-fallback warning) is not fixed: it keeps its old rollback, and so do the shared helper-local var series of a script that has such a payload.
  • Over the 555 corpus and fixture scripts the emitted C++ differs from the parent in 72, only inside restore_script_state; the other 483 are byte-identical.
  • One documented corner: a requested TA whose length is a var input is re-created by the chart's first-bar block, as before (it writes the object and touches no history or map).
  • The fix does not depend on which fill routes can reach a late restore; it removes the loss at every restore.

Why the restore is safe to skip

The pre-body restore/feed/snapshot bracket and the closing restores only ever rolled back chart state; nothing replays a requested feed on the premise that a restore rewound it, and run preparation resets every declaration independently of the restore inventory.

Tests

  • A lab witness (the unchanged first commit) fails on the parent with the counter at 11 against 13 and passes with the change.
  • A second witness has the chart read offset history of a requested series across the same late restore.
  • A static test asserts, on six payload shapes in both ordinary and handle modes plus tuple and lower-timeframe requests, that requested progression is copied but not restored, that result cells and chart members are restored, that chart-fallback members stay restored, and that no skipped member is mentioned by chart-side code.
  • Two pinned emission hashes and the whole-C++ identity test now compare with the rollback hook body removed; the hashes of the parent and of this change agree under that comparison.

Full suite on the engine main: no failures. A parity sweep on the pushed head is attached below.

🤖 Generated with Claude Code

luisleo526 and others added 4 commits October 26, 2026 18:00
…est.security evaluation

A lab probe built only from shipped features: calc_on_order_fills, two
same-symbol request.security calls (30m lookahead off, 5m lookahead on) whose
helper keeps a var counter, and a late margin-call liquidation after the
script walk. The requested counters are compared with a pure-Python count of
completed requested buckets on every bar, and with a control run without the
margin call. It asserts that the stressed run really executed the margin fill
and the restore. It fails on the current tree: the counter reads 11 where the
feed gives 13.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ss a late restore

tests/test_security_coof_restore_members.py transpiles scripts with a
finer lookahead request (numeric and string helper state, TA binding
variants, OHLC and TA histories with the bar-index clock, mutable globals,
expression and session histories) and asserts on the emitted rollback hook:
requested-progression members are copied but not restored, result cells and
chart members are restored, a chart-fallback payload keeps its members in the
restore, and tuple and lower-timeframe result cells still restore. Every
expectation is derived from the emitted code of the same script.

tests/test_e2e_security_coof_restore_history_reads.py runs the chart reading
offset history of a requested series across the late margin restore and
compares with a feed-derived oracle.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…-evaluation state

Under calc_on_order_fills the engine commits the script-state checkpoint
before it feeds the deferred requested evaluations of a chart bar. A later
fill recalculation (for example a late margin call) restored that older
checkpoint, so the generated state of the request.security context (helper
series and their var seeds, requested TA objects, OHLC and expression
histories, mutable copies, bar-index clocks) lost the deferred writes while
the engine's requested aggregators kept advancing: a silent wrong value.

The generated restore_script_state() now skips exactly those members, taken
from the same metadata that declares them, and still restores chart state and
the merged request results. Snapshot and commit are unchanged. A payload that
keeps user calls on the chart's bar (the chart-fallback warning) keeps its old
rollback, and so do the shared helper maps of a script that has one. No engine
or ABI change. Over the 555 corpus and fixture scripts the emitted C++ differs
from the parent revision in 72 of them, and only inside that one function.

Pins that hash the whole emission now hash it with the rollback hook body
removed, so they keep proving that everything else is unchanged:
- map source-order script 1: old (full emission) 8c6ee0d0..., new (hook body
  removed) 25368130..., reason: the hook body differs by design; the stripped
  emission of the parent revision gives the same 25368130...
- map source-order script 2: old acd85265..., new 074bda25..., same reason and
  the same equality with the parent revision.
The whole-C++ identity test against the parent compares with the hook body
removed on both sides (shared helper without_restore_script_state).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the 1.8.0 release notes. The release note for this fix is a
separate 1.8.1 (planned) section above the planned-1.8.0 section.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 26, 2026
…ialization, request.currency_rate, helper-local var color, one-line field assignment) (#261)

## What

This branch carries five codegen fixes planned for 1.8.0. It is five merge commits on `main`, one per pull request, in this order, followed by one CHANGELOG commit.

| Merged pull request | Fix |
|---|---|
| #260 | A `calc_on_order_fills` recalculation after a fill no longer rolls back the state of a deferred `request.security` evaluation (a silent wrong value, for example after a late margin-call liquidation). |
| #259 | A persistent (`var`) UDT, array, map or matrix whose initializer reads an earlier computed value is initialized once, in statement order (Fixes #226). A `varip` declaration with the same dependency still reads the default; that gap is tracked in #258. |
| #254 | `request.currency_rate` raises its error only where the call is evaluated, not where it merely appears in a branch that never runs (Fixes #228). |
| #255 | `var color` state inside a `request.security` helper, explicit or inferred, with persistence, transparency and `na` (Part of #229). The refusal for any other unsupported helper-local `var` type has the new code `PF-E3093`. |
| #253 | A UDT field assignment as the body of a one-line function or method (Part of #230). |

The CHANGELOG commit puts all five entries in the planned 1.8.0 section, updates its summary and compatibility notes, and removes the separate planned-patch section one of the merged branches added.

## Scope and limits

- The five merge commits resolved no code conflict; the only textual conflicts were in `CHANGELOG.md`, replaced once by the last commit. The name-status against `main` is the union of the five pull requests' own files (28 files).
- Over the 555 corpus and fixture scripts the emitted C++ differs from `main` in 72 scripts, all from the #260 change and only inside `restore_script_state()`; the other 483 are byte-identical. The other four fixes change no script of the 555; their tests cover their shapes.
- The user-facing text of two diagnostics (`PF-E1088`, `PF-W1088`) is reworded and one code is added (`PF-E3093`); the catalog schema is unchanged.
- The second half of #229 (the history of a derived helper argument) and the `varip` gap (#258) are not in this branch; they follow as separate pull requests.

## Tests

- Full suite on the head of this branch with the paired engine: 10725 passed, 221 skipped (private fixtures and empty parameter sets, the same 221 as `main`), 0 failed.
- A parity sweep over the whole population (7989 graded probes): PASS, no regression and no improvement (hard gate: 1009 evaluated, no moves; coverage complete).
- Every pull request above keeps the tests it was reviewed with; this branch adds no test of its own.

Fixes #226, Fixes #228. Part of #229, Part of #230.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@luisleo526

Copy link
Copy Markdown
Contributor Author

Superseded by #261, which carried this fix into the 1.8.0 release.

@luisleo526 luisleo526 closed this Oct 26, 2026
luisleo526 added a commit that referenced this pull request Oct 26, 2026
… bound to a derived expression (#262)

## What

`request.security` now accepts a helper that reads the history of a parameter bound to a derived expression, as TradingView does. The reporter's script from #229 translates:

```pine
f(x) => x[2] - x
a = ta.sma(close, 3)
b = f(a)
c = request.security(syminfo.tickerid, '60', b)
```

Before, the transpiler refused it ("helper-parameter history currently requires a direct OHLC/time bar-series binding"). The helper's `x[n]` read the chart's bars; it now reads the argument's own values on the requested context's bars.

This is the second half of #229. The first half (`var color` helper state) is in #261.

## How it works

- Each call site that needs the history keeps its own ring of argument values, sampled on the requested context's bars, so `x[n]` returns the value the argument had `n` evaluations of that call ago.
- Admission is by the position of the call. Direct OHLC and time bindings, builtin forms and method envelopes keep their existing lowering.
- The ring is requested-evaluation state. The restore after a `calc_on_order_fills` fill (#260, in #261) leaves it alone, like the rest of the requested progression state.
- The history of a global that is fully overwritten before the request statement replays its last unconditional write.

## Refusals

Shapes that cannot be sampled are refused with located errors, `PF-E3094` through `PF-E3101`: a derived history offset that is not a non-negative constant integer, a ring capacity that is not representable, a call that is not unconditional, a helper that depends on other helpers' state, history composed with helper-result history, a mutable global that cannot be replayed, and a parameter whose type or width has no lossless history. The code numbers follow `PF-E3093` from #261; messages are unchanged by the numbering.

## Scope and limits

- Over the 555 corpus and fixture scripts the emitted C++ is identical to main: none of them contains a derived helper-argument history, so the change reaches only scripts of the shape above (the tests cover them).
- One diagnostic code range is added (`PF-E3094` to `PF-E3101`); the catalog schema is unchanged. The CHANGELOG entry is in the 1.8.0 section.

## Tests

- Full suite with the paired engine: 10837 passed, 221 skipped (private fixtures and empty parameter sets, the same 221 as main), 0 failed. The branch adds the spec test of the reporter's script (red on main), a compiled-and-run witness of the sampled values, a refusal test with one row per shape, and a witness that the ring survives a late fill recalculation.

Fixes #229

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant