Skip to content

Commit 288f189

Browse files
authored
Exact tick-built bar volume and a genuine-tick equivalence harness (#319)
* Qualify genuine tick tapes against realtime and modeled references * Accumulate tick volume on decimal quantity grids A bar built from ticks must equal the same bar built offline from the same prints. Keep a Neumaier fallback for off-grid quantities and integer overflow. Hash all tick-volume accumulator state; tick stream hashes change forward-only. Confirmed-bar batch accumulation remains unchanged. * Re-anchor consumer documentation after tick-volume changes * Preserve legacy hashes without tick-volume precision state A bar built from ticks must equal the same bar built offline from the same prints. Omit default and redundant additive state from the tick-volume hash extension; retain every accumulator field when integer-grid or compensated state adds information. Preserve existing tick pins as well as bar and batch hashes.
1 parent 700c5d2 commit 288f189

15 files changed

Lines changed: 1437 additions & 81 deletions

‎docs/adr/0001-kernel-adapter-boundary.md‎

Lines changed: 11 additions & 11 deletions
Large diffs are not rendered by default.

‎docs/design/native-feature-parity.md‎

Lines changed: 49 additions & 49 deletions
Large diffs are not rendered by default.

‎docs/native-refactor-progress.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -194,9 +194,9 @@ them for the record only. "Linux aarch64" is the maintainers' Cortex-A725 host.
194194
K4, the batch log pre-size. Since V19-B's step 6 (`08e71bae`) a run under the
195195
default `NativeEventRetention::Window` keeps no driver point and retires its
196196
journal at every script-bar boundary, so K4 has nothing to size.
197-
`reserve_driver_log` (`src/native_execution_consumer.cpp:2707-2712`) returns
197+
`reserve_driver_log` (`src/native_execution_consumer.cpp:2733-2738`) returns
198198
unless the retention is Full, and `presize_logs`
199-
(`src/native_execution_consumer.cpp:9440-9451`) sizes no journal under Window.
199+
(`src/native_execution_consumer.cpp:9460-9471`) sizes no journal under Window.
200200
`NativeRunSpec::event_retention` (`include/pineforge/native_run_spec.hpp:714`)
201201
defaults to Window, and the Pine adapter declares
202202
`NativeEventRetention::Window` (`src/source/pine_adapter.cpp:2494`). K4 still

‎docs/pages/native-engine.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -638,13 +638,13 @@ a host reacts to its own execution and may submit again. A request born there,
638638
mid-bar on a continuous segment, is eligible on the **remaining path suffix** of
639639
that segment — the birth is admitted at the current cursor and the geometric
640640
search then sees only the unconsumed suffix (`born_on_remaining_path`,
641-
`native_execution_consumer.cpp:5765-5769`). Requests accepted before the
641+
`native_execution_consumer.cpp:5791-5795`). Requests accepted before the
642642
segment, and discrete points, keep the ordinary birth gate above.
643643

644644
`on_native_bar_open` fires at the modeled opening, before that point's matching
645-
pass (`native_execution_consumer.cpp:7220-7222`). **Lookahead warning:** the
645+
pass (`native_execution_consumer.cpp:7247-7249`). **Lookahead warning:** the
646646
`Bar` it receives is the *complete* script bar — the consumer has already set
647-
`engine.current_bar_ = open_view` (`native_execution_consumer.cpp:7214`), the
647+
`engine.current_bar_ = open_view` (`native_execution_consumer.cpp:7221`), the
648648
complete bar unless the spec asks for `NativeOpenBarView::OpenOnly` — so its
649649
high, low and close are the finished bar's, not what is known at the open. A
650650
host that must decide on open-only information reads
@@ -2112,7 +2112,7 @@ default, set while no run is active — because each row is a full
21122112
the live state, not the run's length: the closed rows enter through a running
21132113
digest). With the switch on,
21142114
one row follows each point, after the extremes that point just folded
2115-
(`record_script_report_point`, `native_execution_consumer.cpp:8102`), so
2115+
(`record_script_report_point`, `native_execution_consumer.cpp:8129`), so
21162116

21172117
```text
21182118
broker_state_hash_len == equity_curve_len == script_bars_processed
@@ -3086,7 +3086,7 @@ Only completed buckets are published, so this recipe has no lookahead by
30863086
construction. It is the same class the kernel's own subscription evaluator
30873087
aggregates with, and the one the kernel's `script_bucket_completions` query
30883088
feeds when the Pine scheduler asks how its input span buckets
3089-
(`TimeframeAggregator` `native_execution_consumer.cpp:8184`). What it does
3089+
(`TimeframeAggregator` `native_execution_consumer.cpp:8211`). What it does
30903090
**not** give you is what a
30913091
declared subscription does: an `authoritative_bars` feed, the `gaps` and
30923092
`lookahead` delivery rules, the lazy-seal chronology, a C spelling, and the

‎docs/pages/pine-to-native.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -127,8 +127,8 @@ rather than defaulted.
127127
| `commission_type` | `NativeFeeKind` native_run_spec.hpp:19 | `fee_kind` pineforge.h:502 | `native_open_lots_strategy.cpp` | `Percent`, `CashPerUnit`, `CashPerExecution` — the same three TradingView offers. |
128128
| `commission_value` | `fee_value` native_run_spec.hpp:658 | `fee_value` pineforge.h:502 | `native_open_lots_strategy.cpp` | Quoted per execution by `quote_execution_commissions` engine_execution.cpp:1588. A cash-per-execution ticket is split by units over every slice of that execution. |
129129
| `slippage` | `slippage_ticks` native_run_spec.hpp:652 | `slippage_ticks` pineforge.h:502 | `native_price_grid_strategy.cpp` | Raw `ticks * price_tick`, applied once native_execution_consumer.cpp:3980-3981. TradingView's round-then-slip order is adapter-only. |
130-
| `process_orders_on_close` | `NativeCloseExecution` native_run_spec.hpp:33 | `close_execution` pineforge.h:503 | `native_open_lots_strategy.cpp` | `AfterCalculation` emits the post-calculation close point native_execution_consumer.cpp:7786-7787, so a request submitted in bar *k*'s calculation fills at bar *k*'s own close. |
131-
| `pyramiding` | `max_open_lots` native_run_spec.hpp:664 | `max_open_lots` pineforge.h:505 | `native_open_lots_strategy.cpp` | Caps surviving+new physical lots (`resulting_lot_count` native_execution_consumer.cpp:3096). The Pine adapter keeps its own per-cycle *entry count*, ruled **adapter-policy** in ADR-0001's table row `max_open_lots`; measured against TradingView it is not TradingView's rule, and this cap is the closer one (14 of 15 tape scenarios against 13, `tests/test_pyramiding_count_differential.cpp`). Lowered onto the cap, four corpus probes move away from TradingView, for want of its first-eligible-point rule, so the adapter keeps its count. |
130+
| `process_orders_on_close` | `NativeCloseExecution` native_run_spec.hpp:33 | `close_execution` pineforge.h:503 | `native_open_lots_strategy.cpp` | `AfterCalculation` emits the post-calculation close point native_execution_consumer.cpp:7813-7814, so a request submitted in bar *k*'s calculation fills at bar *k*'s own close. |
131+
| `pyramiding` | `max_open_lots` native_run_spec.hpp:664 | `max_open_lots` pineforge.h:505 | `native_open_lots_strategy.cpp` | Caps surviving+new physical lots (`resulting_lot_count` native_execution_consumer.cpp:3122). The Pine adapter keeps its own per-cycle *entry count*, ruled **adapter-policy** in ADR-0001's table row `max_open_lots`; measured against TradingView it is not TradingView's rule, and this cap is the closer one (14 of 15 tape scenarios against 13, `tests/test_pyramiding_count_differential.cpp`). Lowered onto the cap, four corpus probes move away from TradingView, for want of its first-eligible-point rule, so the adapter keeps its count. |
132132
| `default_qty_type` | `Sized` native_order.hpp:164 with `CashValue` native_order.hpp:91 or `EquityFraction` native_order.hpp:97 | `PF_NATIVE_INTENT_SIZED` native_c_api.h:348 with `size_basis` native_c_api.h:2147 | `native_sized_report_strategy.cpp` | The size is a property of the *request*, not of the run: a native host may size one order by cash and the next by equity fraction. `Transact` native_order.hpp:41 is the fixed-units form. Pine itself has no per-call `qty_type` — TradingView's compiler refuses one — so a script's `strategy.entry(qty = ...)` is always units; the source layer's typed per-call entry, which only generated or C++ callers reach, converts its money exactly as the declared default does. |
133133
| `default_qty_value` | the basis's own scalar: `CashValue::cash` native_order.hpp:92, `EquityFraction::fraction` native_order.hpp:98 | `intent_value` native_c_api.h:2151 | `native_sized_report_strategy.cpp` | `SizeTime` native_order.hpp:106 chooses *when* the basis resolves and `SizePrice` native_order.hpp:146 *which* price it converts at; both compose. |
134134
| `calc_on_order_fills` | `NativeCalculationTrigger` native_run_spec.hpp:94 | `calculation` native_c_api.h:2321 | `native_calc_on_fills_strategy.cpp` | `BarCloseAndFills` recalculates once at the cursor of each applied execution, bounded by `max_recalculations_per_point` native_run_spec.hpp:695. The generic cadence only: TradingView's waypoint-only refill, its two-fills-at-open rule and its script-state rollback stay in the adapter. |
@@ -296,7 +296,7 @@ is `strategy_native_submit_v1` native_c_api.h:2723 over a
296296
| `strategy.oca.reduce` | `GroupEffect::Reduce` native_order.hpp:466 | `PF_NATIVE_GROUP_REDUCE` native_c_api.h:479 | `tests/test_native_resting_group_contract.cpp` | The sibling's quantity is reduced instead of cancelled. |
297297
| `strategy.oca.none` | `NoGroup` native_order.hpp:467 | `PF_NATIVE_GROUP_NONE` native_c_api.h:472 | `hello_kernel.cpp` | The default: no group, no sibling effect. |
298298
| `strategy.direction.all` | `NativeOpenDirections::Both` native_run_spec.hpp:76 | `allowed_open_directions` pineforge.h:503 | `native_risk_limits_strategy.cpp` | The default. |
299-
| `strategy.direction.long` | `NativeOpenDirections::Long` native_run_spec.hpp:74 | `allowed_open_directions` pineforge.h:503 | `tests/test_native_run_spec.cpp` | An opening the direction forbids is refused with `OpeningDirection` native_execution_consumer.cpp:3085. |
299+
| `strategy.direction.long` | `NativeOpenDirections::Long` native_run_spec.hpp:74 | `allowed_open_directions` pineforge.h:503 | `tests/test_native_run_spec.cpp` | An opening the direction forbids is refused with `OpeningDirection` native_execution_consumer.cpp:3111. |
300300
| `strategy.direction.short` | `NativeOpenDirections::Short` native_run_spec.hpp:75 | `allowed_open_directions` pineforge.h:503 | `tests/test_native_run_spec.cpp` | Same, the other way. |
301301
| `strategy.commission.percent` | `NativeFeeKind::Percent` native_run_spec.hpp:20 | `fee_kind` pineforge.h:502 | `native_sized_report_strategy.cpp` | `fee_value` is percent ÷ 100 of the absolute account notional. |
302302
| `strategy.commission.cash_per_contract` | `NativeFeeKind::CashPerUnit` native_run_spec.hpp:21 | `fee_kind` pineforge.h:502 | `tests/test_native_selected_requests.cpp` | Account currency per unit. |
@@ -405,7 +405,7 @@ block is the `PF_NATIVE_SPEC_EXT_RISK` native_c_api.h:739 tail.
405405

406406
| Pine | C++ | C | Runs in | Notes |
407407
| --- | --- | --- | --- | --- |
408-
| `strategy.risk.allow_entry_in()` | `allowed_open_directions` native_run_spec.hpp:665 | `allowed_open_directions` pineforge.h:503 | `native_risk_limits_strategy.cpp` | The one `strategy.risk.*` rule that is already the kernel's, per ADR-0001's `risk` ruling row. Rejects with `OpeningDirection` native_execution_consumer.cpp:3085. |
408+
| `strategy.risk.allow_entry_in()` | `allowed_open_directions` native_run_spec.hpp:665 | `allowed_open_directions` pineforge.h:503 | `native_risk_limits_strategy.cpp` | The one `strategy.risk.*` rule that is already the kernel's, per ADR-0001's `risk` ruling row. Rejects with `OpeningDirection` native_execution_consumer.cpp:3111. |
409409
| `strategy.risk.max_position_size()` | `max_abs_units` native_run_spec.hpp:663 | `max_abs_units` pineforge.h:504 | `tests/test_native_run_spec.cpp` | Tests the *resulting* book. TradingView gates the *live* book before the fill, so an entry is refused only once the book already holds the limit — ruled **adapter-policy** in ADR-0001's `max_abs_units` row, with lane N12's measurement (the adapter ends at 4 where the kernel cap ends at 2). |
410410
| `strategy.risk.max_drawdown()` | `NativeRiskLimits::max_drawdown` native_run_spec.hpp:311 | `risk_max_drawdown` native_c_api.h:2357 | `native_risk_limits_strategy.cpp` | `NativeLossLimit` native_run_spec.hpp:259 is account currency, or a percent of the running peak when `percent` native_run_spec.hpp:261 is set. Blocks openings to the end of the run. |
411411
| `strategy.risk.max_intraday_loss()` | `NativeRiskLimits::max_intraday_loss` native_run_spec.hpp:312 | `risk_max_intraday_loss` native_c_api.h:2360 | `native_risk_limits_strategy.cpp` | Percent is of the day's opening equity. Blocks to the end of its own day. |

‎src/native_execution_consumer.cpp‎

Lines changed: 128 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1766,6 +1766,30 @@ uint64_t NativeExecutionConsumer::continuation_hash() const noexcept {
17661766
f.b(script_.modeled_ohlc);
17671767
f.b(has_forming_);
17681768
if (has_forming_) hash_bar(f, forming_);
1769+
const auto has_tick_volume_extension = [](const TickVolume& volume, double published_volume) {
1770+
return volume.units != 0 || (volume.sum != 0.0 && volume.sum != published_volume)
1771+
|| volume.compensation != 0.0 || volume.divisor != 1.0
1772+
|| volume.decimal_places != 0 || volume.exact;
1773+
};
1774+
if (input_mode_ == InputMode::ObservedTicks
1775+
&& (has_tick_volume_extension(forming_tick_volume_, has_forming_ ? forming_.volume : 0.0)
1776+
|| has_tick_volume_extension(partial_tick_volume_, partial_has_ ? partial_.volume : 0.0))) {
1777+
const auto hash_volume = [&f](const TickVolume& volume) {
1778+
f.i(volume.units);
1779+
f.d(volume.sum);
1780+
f.d(volume.compensation);
1781+
f.d(volume.divisor);
1782+
f.i(volume.decimal_places);
1783+
f.b(volume.exact);
1784+
};
1785+
hash_volume(forming_tick_volume_);
1786+
hash_volume(partial_tick_volume_);
1787+
f.b(partial_has_);
1788+
if (partial_has_) {
1789+
hash_bar(f, partial_);
1790+
f.i(partial_script_open_ms_);
1791+
}
1792+
}
17691793
f.b(has_last_price_);
17701794
f.d(last_price_);
17711795
f.i(last_print_time_ms_);
@@ -2406,6 +2430,8 @@ bool NativeExecutionConsumer::begin_ready(BacktestEngine& engine, NativeRunPhase
24062430
has_tick_sequence_ = false;
24072431
script_ = ScriptBucket{};
24082432
has_forming_ = false;
2433+
forming_tick_volume_ = TickVolume{};
2434+
partial_tick_volume_ = TickVolume{};
24092435
has_last_price_ = false;
24102436
// Stream print state is run-scoped: a reused host must not carry the
24112437
// previous run's last print into the next run's decision coordinates
@@ -6895,6 +6921,7 @@ void NativeExecutionConsumer::clear_partial() noexcept {
68956921
partial_has_ = false;
68966922
partial_ = Bar{};
68976923
partial_script_open_ms_ = 0;
6924+
partial_tick_volume_ = TickVolume{};
68986925
}
68996926

69006927
std::optional<Bar> NativeExecutionConsumer::partial_bar() const {
@@ -9746,6 +9773,78 @@ bool NativeExecutionConsumer::stream_push_bar(BacktestEngine& engine, const Bar&
97469773
}
97479774
}
97489775

9776+
void NativeExecutionConsumer::TickVolume::reset(const std::optional<double>& step) noexcept {
9777+
*this = TickVolume{};
9778+
if (!step) return;
9779+
double scale = 1.0;
9780+
for (int places = 0; places <= 22; ++places, scale *= 10.0) {
9781+
if (*step != 1.0 / scale) continue;
9782+
divisor = scale;
9783+
decimal_places = places;
9784+
exact = true;
9785+
return;
9786+
}
9787+
}
9788+
9789+
double NativeExecutionConsumer::TickVolume::decimal_value(int64_t quantity_units) const noexcept {
9790+
if (quantity_units <= (int64_t{1} << 53))
9791+
return static_cast<double>(quantity_units) / divisor;
9792+
uint64_t numerator = static_cast<uint64_t>(quantity_units);
9793+
uint64_t denominator = 1;
9794+
for (int places = 0; places < decimal_places; ++places) denominator *= 5;
9795+
int exponent = -decimal_places;
9796+
while (denominator <= numerator / 2) {
9797+
denominator *= 2;
9798+
++exponent;
9799+
}
9800+
while (numerator < denominator) {
9801+
numerator *= 2;
9802+
--exponent;
9803+
}
9804+
uint64_t remainder = numerator - denominator;
9805+
uint64_t significand = 1;
9806+
for (int bit = 0; bit < 52; ++bit) {
9807+
remainder *= 2;
9808+
significand *= 2;
9809+
if (remainder >= denominator) {
9810+
remainder -= denominator;
9811+
++significand;
9812+
}
9813+
}
9814+
const uint64_t complement = denominator - remainder;
9815+
if (remainder > complement || (remainder == complement && (significand & 1)))
9816+
++significand;
9817+
return std::ldexp(static_cast<double>(significand), exponent - 52);
9818+
}
9819+
9820+
void NativeExecutionConsumer::TickVolume::add(double quantity) noexcept {
9821+
const double next_sum = sum + quantity;
9822+
if (std::isfinite(next_sum)) {
9823+
compensation += sum >= quantity
9824+
? (sum - next_sum) + quantity : (quantity - next_sum) + sum;
9825+
} else {
9826+
compensation = 0.0;
9827+
}
9828+
sum = next_sum;
9829+
if (!exact) return;
9830+
const double rounded = std::round(quantity * divisor);
9831+
if (!std::isfinite(rounded) || rounded >= std::ldexp(1.0, 63)) {
9832+
exact = false;
9833+
return;
9834+
}
9835+
const int64_t quantity_units = static_cast<int64_t>(rounded);
9836+
if (decimal_value(quantity_units) != quantity
9837+
|| quantity_units > std::numeric_limits<int64_t>::max() - units) {
9838+
exact = false;
9839+
return;
9840+
}
9841+
units += quantity_units;
9842+
}
9843+
9844+
double NativeExecutionConsumer::TickVolume::value() const noexcept {
9845+
return exact ? decimal_value(units) : sum + compensation;
9846+
}
9847+
97499848
bool NativeExecutionConsumer::preflight_ticks(BacktestEngine& engine, const TradeTick* ticks, int n) {
97509849
if (n < 0 || (n > 0 && ticks == nullptr)) {
97519850
present_refusal(engine, "native tick array is invalid");
@@ -9763,11 +9862,15 @@ bool NativeExecutionConsumer::preflight_ticks(BacktestEngine& engine, const Trad
97639862
uint64_t prev_sequence = last_tick_sequence_;
97649863
bool prev_has_sequence = has_tick_sequence_;
97659864
uint64_t ordinals = next_timeline_ordinal_;
9865+
const auto* spec = spec_ptr();
9866+
const auto step = spec ? spec->quantity_grid : std::optional<double>{};
97669867
std::optional<int64_t> volume_slot;
9767-
double volume = 0.0;
9868+
TickVolume volume = forming_tick_volume_;
9869+
std::optional<int64_t> partial_slot;
9870+
TickVolume partial_volume = partial_tick_volume_;
9871+
if (partial_has_) partial_slot = partial_script_open_ms_;
97689872
if (has_forming_) {
97699873
volume_slot = forming_.timestamp;
9770-
volume = forming_.volume;
97719874
}
97729875
int64_t prev_array_ts = 0;
97739876
bool has_array_prev = false;
@@ -9812,10 +9915,17 @@ bool NativeExecutionConsumer::preflight_ticks(BacktestEngine& engine, const Trad
98129915
}
98139916
if (!volume_slot || *volume_slot != interval->open_ms) {
98149917
volume_slot = interval->open_ms;
9815-
volume = 0.0;
9918+
volume.reset(step);
9919+
}
9920+
volume.add(tick.quantity);
9921+
const auto script_interval = script_interval_at(tick.timestamp);
9922+
const int64_t script_open = script_interval ? script_interval->open_ms : interval->open_ms;
9923+
if (!partial_slot || *partial_slot != script_open) {
9924+
partial_slot = script_open;
9925+
partial_volume.reset(step);
98169926
}
9817-
volume += tick.quantity;
9818-
if (!std::isfinite(volume)) {
9927+
partial_volume.add(tick.quantity);
9928+
if (!std::isfinite(volume.value()) || !std::isfinite(partial_volume.value())) {
98199929
present_refusal(engine, "native tick volume overflow");
98209930
return false;
98219931
}
@@ -9896,10 +10006,12 @@ bool NativeExecutionConsumer::finalize_observed_tick_slot(
989610006
}
989710007
}
989810008
has_forming_ = false;
10009+
forming_tick_volume_ = TickVolume{};
989910010
return !failed();
990010011
}
990110012
const Bar formed = forming_;
990210013
has_forming_ = false;
10014+
forming_tick_volume_ = TickVolume{};
990310015
if (!contribute_input(engine, formed, interval, next_interval_index_,
990410016
InputContribution::ObservedTickSlot)) {
990510017
return false;
@@ -9993,7 +10105,13 @@ bool NativeExecutionConsumer::deliver_tick(BacktestEngine& engine, const TradeTi
999310105
calculating_bar_has_ = true;
999410106
// The print is the cursor, and it is real traded activity: the bar so far
999510107
// folds its price and its quantity before the observation hook runs.
9996-
note_partial_point(tick_context.decision.script_bar_open_ms, tick.price, tick.quantity);
10108+
const auto* spec = spec_ptr();
10109+
const auto step = spec ? spec->quantity_grid : std::optional<double>{};
10110+
if (!partial_has_ || partial_script_open_ms_ != tick_context.decision.script_bar_open_ms)
10111+
partial_tick_volume_.reset(step);
10112+
note_partial_point(tick_context.decision.script_bar_open_ms, tick.price, 0.0);
10113+
partial_tick_volume_.add(tick.quantity);
10114+
partial_.volume = partial_tick_volume_.value();
999710115
if (!invoke_tick_callback(engine, tick_bar, tick_context)) {
999810116
processing_input_ = false;
999910117
return false;
@@ -10010,15 +10128,17 @@ bool NativeExecutionConsumer::deliver_tick(BacktestEngine& engine, const TradeTi
1001010128
has_last_price_ = true;
1001110129
last_print_time_ms_ = tick.timestamp;
1001210130
if (!has_forming_) {
10013-
forming_ = Bar{tick.price, tick.price, tick.price, tick.price, tick.quantity,
10131+
forming_tick_volume_.reset(step);
10132+
forming_ = Bar{tick.price, tick.price, tick.price, tick.price, 0.0,
1001410133
interval->open_ms};
1001510134
has_forming_ = true;
1001610135
} else {
1001710136
forming_.high = std::max(forming_.high, tick.price);
1001810137
forming_.low = std::min(forming_.low, tick.price);
1001910138
forming_.close = tick.price;
10020-
forming_.volume += tick.quantity;
1002110139
}
10140+
forming_tick_volume_.add(tick.quantity);
10141+
forming_.volume = forming_tick_volume_.value();
1002210142
last_observed_slot_open_ = interval->open_ms;
1002310143
if (tick.sequence != 0) {
1002410144
last_tick_sequence_ = tick.sequence;

0 commit comments

Comments
 (0)