Skip to content

feat(matrix): typed/UDT matrix support — PineGenericMatrix<T> - #1

Merged
luisleo526 merged 16 commits into
mainfrom
feat/generic-matrix-template
May 10, 2026
Merged

luisleo526 merged 16 commits into
mainfrom
feat/generic-matrix-template

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Summary

Phase 1 of typed matrix support: header-only PineGenericMatrix<T> template alongside existing PineMatrix (Eigen, double-only).

  • Element types: int, bool, string, color, UDT
  • Storage: `std::vector<std::vector>` (general), `std::vector<std::vector>` (bool)
  • Methods: new_, get/set/fill, rows/columns, row/col/row_ref (SFINAE-disabled for bool), add_row/add_col, remove_row/remove_col, swap_rows/swap_columns, copy/submatrix/transpose/concat, reshape/reverse, sort (compile-gated), elements_count
  • Series<PineGenericMatrix> history support
  • Bumps to v0.3.0

Pairs with codegen `feat/typed-matrix-dispatch` (filed separately) which routes `matrix` declarations to this template.

Test plan

  • All 16 generic_matrix tests pass via ctest
  • No regression in existing engine tests
  • Codegen sibling repo verified compatible

luisleo526 and others added 16 commits May 10, 2026 07:56
…SOURCES

Code review issues from spec compliance pass:
- add_col silently read past end of values vector when shorter than rows
  (UB); add_row had similar silent jagged-row hazard. Both now throw on
  size mismatch, matching PineMatrix::add_col parity.
- Test target registered as one-off bypassed TEST_SOURCES foreach loop
  used for coverage instrumentation; moved into TEST_SOURCES.
Add explicit PineGenericMatrix<bool> specialization using
std::vector<std::vector<char>> internal storage to avoid the
vector<bool> proxy footgun. Cast at boundary: bool→char(1/0) on
input, char!=0→bool on output. row_ref intentionally absent.
Add test_generic_matrix_bool_proxy; 19/19 ctest targets pass.
Series<PineGenericMatrix<UDT>> hit a link-time gap because na.hpp only
specialized double/int/int64_t/bool. Provide a body for the primary
template that returns T{} for any default-constructible T; existing
explicit specializations still take precedence for primitive types.

Removes the local na specialization workaround from the Series test
and adds direct fallback coverage in test_na_runtime.
Move matrix<UDT> from 'Not implemented anywhere' to first-class
supported, add PineGenericMatrix<T> module row, and add a 'Typed
matrices' coverage row alongside 'Numeric matrices'.
Adds header-only PineGenericMatrix<T> for matrix<int/bool/string/color/UDT>
alongside existing PineMatrix (Eigen, double-only). C ABI unchanged
(generic matrix is internal compiler-emitted code; never crosses .so
boundary).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@luisleo526
luisleo526 merged commit 6790e75 into main May 10, 2026
4 checks passed
@luisleo526
luisleo526 deleted the feat/generic-matrix-template branch May 10, 2026 15:23
luisleo526 added a commit that referenced this pull request May 12, 2026
Pin the contract that strategy.entry's same-id replacement is a bar-
boundary event: the prev bar's pending order is evaluated against the
new bar's OHLC FIRST, then on_bar runs and any same-id placement
takes effect for the NEXT bar (never the current one).

Four scenarios:
1. Filled stop unaffected by same-bar same-id replace — verifies
   process_pending_orders runs before on_bar so the prev-bar stop
   gets its OHLC evaluation regardless of any modify guard.
2. Unfilled stop replaced for next bar — verifies the new placement
   is what fires next, not the old one.
3. New entry after same-bar fill defers to next bar — verifies the
   engine does NOT add a second process_pending_orders pass that
   would let on_bar's new placements fire on the same bar.
4. Carry capture on flat session start — pins the
   ``tv_carry_qty=0`` contract for the very first strategy.entry
   call (regression test for the ``position_qty_=1.0`` default leak
   that produced qty=2 trade #1 on probe 62).
luisleo526 added a commit that referenced this pull request Sep 23, 2026
…form through one allocator (INT17b)

The sanitizers lane of PR #279 aborted two lane PERF-P4 rows,
test_native_command_history_read and test_adapter_receipts_in_place:

  ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new vs free)
    #1 operator delete(void*, unsigned long) tests/test_native_command_history_read.cpp:66
  allocated by: operator new(unsigned long, std::nothrow_t const&) (asan_new_delete.cpp)

Each test replaced only the plain/array operator new and the plain/sized
operator delete, to count allocations. libstdc++'s temporary buffers
(std::inplace_merge in NativeExecutionConsumer::events_after,
std::stable_sort in PineExecutionAdapter::apply_open_market_admission) take
their memory from the NOTHROW operator new -- left as ASan's own -- and give
it back through the SIZED operator delete, which the test had routed to
free(). Two allocators in one executable; ASan stops at the mismatch.

tests/global_allocation_replacement.hpp is now the one replacement set: all
twenty replaceable forms (plain, array, nothrow, sized, aligned and aligned
nothrow, for new and for delete) go to one allocator, std::malloc
(posix_memalign past the fundamental alignment) released by std::free, with
the standard new-handler loop and nothrow semantics. A test reads and sets
plain state: `allocations` (every allocation, measured as a difference
around a span), `refusing` / `refused` (every form fails the way it fails,
nothing allocated), and, opted in with PINEFORGE_TEST_ALLOCATION_BYTES,
per-block size headers for a recording window (`recording`, `live_bytes`,
`on_recorded`).

Every test that replaced the global functions now includes it instead of
its own partial set -- all of them were incomplete (no aligned forms; most
had no nothrow forms either), so a libstdc++ path through the missing forms
would have mismatched the same way:

  test_native_command_history_read     P4   counted, was plain+sized only
  test_adapter_receipts_in_place       P4   counted, was plain+sized only
  test_native_calendar_memo_cost       K1   counted, no aligned forms
  test_native_batch_log_presize        K24  byte window, no aligned forms
  test_native_order_resting_core            counted, partial aligned/nothrow
  test_native_selected_core                 counted, plain+sized only
  test_l10ae_allowance_refresh_liveness     counted, plain+sized only
  test_l8b_pending_mirror                   refused, no aligned forms
  test_market_admission_causality           refused, plain+sized only
  test_native_run_spec                      refused, plain(+sized) only
  test_pending_order_mirror_no_alloc_l4d    refused, plain+sized only; its
                                            base body keeps its two names,
                                            now references to the shared state

The uncompiled legacy base tests/test_pending_order_mirror_no_alloc.cpp
(REMOVE_ITEM'd from TEST_SOURCES; its ab9714b body is the twin's parity
reference) is left as it is. No assertion, bound or pinned value changes:
the counted windows read the same numbers (calendar memo cost: canonical
159, tolerant 88, 0.02 extra per bar; batch presize: 22522624 held for
22176000 recorded, 1.016x; the P4 rows 57356 and 98 checks), and nothing
under src/ or include/ changes.

Proof, the sanitizers profile reproduced on spark (Linux aarch64, GCC
13.3.0-6ubuntu2~24.04.1): ci_verify's own configure argv and ctest flags,
the six ABI providers prepared, the job's ASAN_OPTIONS/UBSAN_OPTIONS.
  before (f71cd82): both rows abort with the report above, exactly as CI;
  after, same options: the other ten converted rows pass, including
    test_adapter_receipts_in_place (96.6 s);
  the whole suite, with detect_stack_use_after_return=0 added (aarch64
  GCC 13's fake stacks cost 30-130x per row on that host: 15.56 s against
  0.12 s for one row that CI's x86 runs in 0.51 s):
    before  99% tests passed, 2 tests failed out of 588 (the two rows)
    after  100% tests passed out of 588
Mac release, debug and kernel builds: every converted row passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Sep 23, 2026
…form through one allocator (INT17b)

The sanitizers lane of PR #279 aborted two lane PERF-P4 rows,
test_native_command_history_read and test_adapter_receipts_in_place:

  ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new vs free)
    #1 operator delete(void*, unsigned long) tests/test_native_command_history_read.cpp:66
  allocated by: operator new(unsigned long, std::nothrow_t const&) (asan_new_delete.cpp)

Each test replaced only the plain/array operator new and the plain/sized
operator delete, to count allocations. libstdc++'s temporary buffers
(std::inplace_merge in NativeExecutionConsumer::events_after,
std::stable_sort in PineExecutionAdapter::apply_open_market_admission) take
their memory from the NOTHROW operator new -- left as ASan's own -- and give
it back through the SIZED operator delete, which the test had routed to
free(). Two allocators in one executable; ASan stops at the mismatch.

tests/global_allocation_replacement.hpp is now the one replacement set: all
twenty replaceable forms (plain, array, nothrow, sized, aligned and aligned
nothrow, for new and for delete) go to one allocator, std::malloc
(posix_memalign past the fundamental alignment) released by std::free, with
the standard new-handler loop and nothrow semantics. A test reads and sets
plain state: `allocations` (every allocation, measured as a difference
around a span), `refusing` / `refused` (every form fails the way it fails,
nothing allocated), and, opted in with PINEFORGE_TEST_ALLOCATION_BYTES,
per-block size headers for a recording window (`recording`, `live_bytes`,
`on_recorded`).

Every test that replaced the global functions now includes it instead of
its own partial set -- all of them were incomplete (no aligned forms; most
had no nothrow forms either), so a libstdc++ path through the missing forms
would have mismatched the same way:

  test_native_command_history_read     P4   counted, was plain+sized only
  test_adapter_receipts_in_place       P4   counted, was plain+sized only
  test_native_calendar_memo_cost       K1   counted, no aligned forms
  test_native_batch_log_presize        K24  byte window, no aligned forms
  test_native_order_resting_core            counted, partial aligned/nothrow
  test_native_selected_core                 counted, plain+sized only
  test_l10ae_allowance_refresh_liveness     counted, plain+sized only
  test_l8b_pending_mirror                   refused, no aligned forms
  test_market_admission_causality           refused, plain+sized only
  test_native_run_spec                      refused, plain(+sized) only
  test_pending_order_mirror_no_alloc_l4d    refused, plain+sized only; its
                                            base body keeps its two names,
                                            now references to the shared state

The uncompiled legacy base tests/test_pending_order_mirror_no_alloc.cpp
(REMOVE_ITEM'd from TEST_SOURCES; its ab9714b body is the twin's parity
reference) is left as it is. No assertion, bound or pinned value changes:
the counted windows read the same numbers (calendar memo cost: canonical
159, tolerant 88, 0.02 extra per bar; batch presize: 22522624 held for
22176000 recorded, 1.016x; the P4 rows 57356 and 98 checks), and nothing
under src/ or include/ changes.

Proof, the sanitizers profile reproduced on spark (Linux aarch64, GCC
13.3.0-6ubuntu2~24.04.1): ci_verify's own configure argv and ctest flags,
the six ABI providers prepared, the job's ASAN_OPTIONS/UBSAN_OPTIONS.
  before (f71cd82): both rows abort with the report above, exactly as CI;
  after, same options: the other ten converted rows pass, including
    test_adapter_receipts_in_place (96.6 s);
  the whole suite, with detect_stack_use_after_return=0 added (aarch64
  GCC 13's fake stacks cost 30-130x per row on that host: 15.56 s against
  0.12 s for one row that CI's x86 runs in 0.51 s):
    before  99% tests passed, 2 tests failed out of 588 (the two rows)
    after  100% tests passed out of 588
Mac release, debug and kernel builds: every converted row passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Sep 24, 2026
… new/delete form through INT17b's one allocator

Four slice-B rows defined their own partial set of the replaceable global
allocation functions to count heap allocations -- the pattern INT17b found
aborting under the Linux sanitizers job (alloc-dealloc-mismatch: libstdc++'s
temporary buffers take the NOTHROW operator new, left as ASan's, and give it
back through a SIZED operator delete the test had routed to free()):

  test_native_intrabar_scratch      PERF-L1  plain/array/nothrow new, plain/sized delete
  test_adapter_quiet_bar (+ its     PERF-L4  plain/array new, plain/sized delete
    _probe and _ungated builds)
  test_adapter_command_allocations  PERF-L5  plain/array new, plain/sized delete
  test_native_match_allocations     PERF-L5  plain/array new, plain/sized delete

Each now includes tests/global_allocation_replacement.hpp (all twenty forms,
one malloc/free allocator) instead:

  * intrabar_scratch, command_allocations, match_allocations take the
    difference of global_allocation::allocations around the span they
    counted before (the run; the one re-issued strategy.exit call; the
    window between script bars kFirstCounted and kLastCounted), the
    style of INT17b's counted conversions; their on/off flag had no other
    reader and is gone.
  * quiet_bar keeps its body: `allocations` is now a reference to the
    shared count (the _l4d twin's precedent), reset where it was reset,
    and `count_allocations` still says whether a bar records its count.
    Every recorded span lies inside the run it was counted over, so each
    per-bar entry is the count it was.

Nothing under src/ or include/ changes (engine code has no over-aligned
allocation, so the aligned forms add nothing to a count). The four rows
print exactly what they printed before the change, line for line, on the
Mac release build: intrabar 0.003-0.045 allocations/bar per path, quiet
bars 4 per stretch (nothing live: 4) magnifier off and on, the re-issued
exit 301 allocations over 42 calls, the matcher 0 per window for every
book. No assertion, bound or pinned value changes.

Fail-before, the sanitizers profile on spark (Linux aarch64, GCC 13.3,
ci_verify's own configure argv, the job's ASAN_OPTIONS/UBSAN_OPTIONS plus
detect_stack_use_after_return=0): the four TUs as the lanes wrote them,
compiled with the sanitizer build's own compile and link commands --

  test_adapter_quiet_bar            aborts: AddressSanitizer:
  test_adapter_command_allocations    alloc-dealloc-mismatch (operator new vs free)
      #1 operator delete(void*, unsigned long)  <the test's own sized delete>
      #2 std::__detail::__return_temporary_buffer<PendingSameBarCommand>
      #5 std::stable_sort  in PineExecutionAdapter::flush_pending_same_bar_commands
      allocated by: operator new(unsigned long, std::nothrow_t const&) (asan)
  test_native_intrabar_scratch      passes (8 passed)
  test_native_match_allocations     passes (ok)

-- the mechanism INT17b fixed, so both adapter rows would have failed CI's
sanitizers job; the two kernel rows are converted so no counting test is
left with a partial set. After: all four pass there. A grep of tests/ for
global operator new/delete definitions now finds only the helper and the
uncompiled legacy base tests/test_pending_order_mirror_no_alloc.cpp that
INT17b left as it is.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526 added a commit that referenced this pull request Sep 26, 2026
…blings its leg had (R5 lane INT26, round 2: the gate sweep's leaver)

INT26's gate sweep (exp-r5-int26-20260926 against rebaseline e69aa1f7)
refused the merge on one leaver,
scrapper:data-BINANCE-BTCUSDT/standard/vasudevshenoy-manoj-betrayed-me
(BTCUSDT 15m, process_orders_on_close, pyramiding 0; stop entries
'Buy'/'Sell' with strategy.exit stop/limit brackets, a pending entry
cancelled when the next bar does not trigger it): 821 of TradingView's 821
trades on main 0d76a09, 820 on ebe6748 (canonical 100 -> 99.9).

Bisect (the probe's own feed, spark, one build per pick boundary): a6e6a00
and 2922ade book 821, e1d7a98 -- R5 lane PAR-ORDERS, H-MEASURE Finding 6b,
a process_orders_on_close stop entry its placing close already reached
fills at that close -- books 820, and so does every later pick.

Mechanism, on 2025-08-03 10:00 UTC. The 09:45 close cancels its pending
'Buy', places it again (stop 113689.29) and re-issues 'Buy Exit' (limit
113771.41); the re-issued legs keep their older book slot, so on the 10:00
bar the pre-script drain settles the limit on its parent's touch bar --
after that bar's receipt read. The 10:00 bar closes at its high 113771.47;
its body cancels 'Buy' again and places it at that high with a fresh 'Buy
Exit' (limit 114004.53), and since e1d7a98 the close pass fills that
re-entry at the close, as TradingView does (#274), so the fresh legs are
working when 10:15 opens. The first limit's terminal receipt is read only
then, and observe_terminal_receipts' sibling cancel withdrew every working
leg of the same exit id, from_entry and origin -- the fresh bracket's
included, both legs at origin 0. The re-entry ran on to the 15:15 session
close and swallowed TradingView's #275 (11:45). Before e1d7a98 the
re-entry rested to 10:15, its legs still staged, so the late read found no
sibling to take. The pick is right; the sibling cancel was not.

Fix, adapter only: cancel_bracket_siblings takes the ordinal the terminal
receipt executed at when the observer reads it, and skips a matching leg of
a later strategy.exit call (another command_sequence) that the kernel
accepted after that ordinal (its Birth::acceptance_ordinal, read through
native_working_requests() only when such a leg is among the matches): that
leg is the same id's later order. A leg of the filled leg's own call is its
sibling however late it was accepted -- the drain submits a bracket's legs
one at a time, executing each, so a stop can be accepted after its limit
filled -- and goes as before (a review of the first form, which skipped
every leg accepted after the fill, found that case). The close passes that
call it synchronously pass nothing and are unchanged. No state is added.

The probe (spark, generated.cpp of codegen a4259656, the lane feed
6b54c44a): 821 trades, canonical rubric excellent, countAbsDelta 0 (820 /
strong before). TradingView's #273-#275 exact: 10:00 113689.29 -> 10:00
113771.41; 10:00 113771.47 -> 11:00 114004.53; 11:45 114005.84 -> 15:15
113844.70. 819 of 821 trades exact to the minute and cent on both legs
(main: 771; the two others, #1 and #3, differ on main too).

Witness: tests/test_pooc_stop_reentry_bracket_tapes.cpp replays three `lab
tv --no-note` tapes of synthetic scripts of the same shape on NYSE:F 15m
(tests/fixtures/pooc_stop_reentry_bracket; tv_trades e0447b53 / b7e89ce1 /
efbd25e3): three flat signal closes that each cancel and place again a stop
entry with its bracket, 10c beyond the close, then 1c (the next bar fills
the entry and then the limit and closes flat), then at that close (filled
there), long on bars 39-41 and short on bars 44-46; TradingView books the
same four rows on the chart and under the magnifier. Every row is compared
field by field, with the end book and its working bracket legs.
- int26-reentry-bracket-chart: all four rows, flat; the short is the
  leaver's shape.
- int26-reentry-bracket-mag records a divergence the fix does not touch
  (the same before and after): there the drain leaves the short's retained
  limit to the flush below the body, whose book still holds the short, so
  the re-entry is never placed -- rows 1-3 exact, both books flat.
- int26-reentry-noexit-chart, the control: the short's re-entry places no
  strategy.exit (TradingView's row 4 still open at the end); the short
  bracket's stop, accepted after its limit filled, must still be withdrawn.
Fail-before: against ebe6748 2 of 27 checks fail (the chart tape: 3 rows
against 4, short 100 against flat); against the first form of the rule 1 of
28 (the control's stop left working); after, 28 of 28.

Floors: one source-bound row. RELEASE_MIN_TESTS 702 -> 703,
EXCLUDED_REGISTERED_MIN 683/683/692 -> 684/684/693 (and
test_ci_verify's expectation, docs/ci.md). KERNEL_MIN_TESTS unchanged.
docs/ci.md's sentence "The PR registration floors at INT26 are 683 for
Debug and sanitizers and 692 for native" is rewritten with the new counts.

Docs: the design page's PAR-ORDERS paragraph records the rule and the
magnified divergence; the CHANGELOG 1.0.0 entry's process_orders_on_close
item adds it for a user. Every doc anchor carried onto this tree by the
exact line map from ebe6748 (135 moved); the three citations of
ci_verify.py's RELEASE_MIN_TESTS line (CONTRIBUTING.md, contributing-llm.md
twice) set by hand, 390 -> 393.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant