Skip to content

engine: Pine arrays stored on the paged object layer, flat in the number of bars - #547

Merged
luisleo526 merged 10 commits into
mainfrom
engine-arrays-paged-store
Oct 27, 2026
Merged

luisleo526 merged 10 commits into
mainfrom
engine-arrays-paged-store

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Summary

Adds the engine's Pine array container, pineforge::ScriptArray<T> (include/pineforge/array.hpp, src/array.cpp). Arrays live in fixed pages over o1::Store with stable ids, so the memory an array retains does not grow with the number of bars, and every public call runs in one transaction so a calc-on-fill rollback restores an array byte for byte.

  • General arrays: fixed pages over o1::Store, stable ids, Store::Tx rollback.
  • Prepared arrays: fixed cardinality is allocated before bar 0, so a bar does no heap allocation.
  • Pine semantics follow the std::vector path the generated code uses today: tolerant negative-index reads give na, the checked v5 and v6 index rules, value copies, nested arrays, drawing-handle arrays and the lower-timeframe clear/append order. There is no capacity refusal and no dropped element.
  • ScriptArrayTransaction hands its handle to the store on every path: rollback() moves the handle out before acting, like commit(), and a failed rollback no longer rolls the scope back a second time or terminates the process.
  • x = y.alias() copies from the borrowed alias and leaves y intact; only an owning array is consumed by a move.

Tests

Registered in tests/cr150.d/p2cx-a5arr.cmake; the 1,000,000-bar growth row is a pinned slow row in scripts/ci_verify.py.

  • growth at 100,000 and 1,000,000 bars against a bound derived from the page size, the header size and the store's shape caps (test_array_growth_100k, test_array_growth_1m)
  • semantics against a vector reference, rollback, nested and handle arrays, preparation, source hooks (test_array_semantics, test_array_rollback, test_array_nested_handles, test_array_preallocated, test_array_source_hooks)
  • each behaviour was also checked by applying a patch that reverts it and confirming its own test fails (11 patches on the first head, plus one for the alias move and one for the transaction wrapper)

Evidence from lab remote ctest-only runs (Release): on the final head, all seven test_array_* tests pass (rj-20261027t064709-c8fa54). Reverting the alias-move fix fails test_array_semantics with array size differs from vector. Reverting the transaction-wrapper fix fails test_array_rollback with array rollback: throwing rollback retained a stale transaction handle (rj-20261027t064713-0c9fe7). The alias-move job on the earlier head was rj-20261027t050111-3f307a.

Not covered

Sanitizers, the Debug and kernel profiles, macOS, the corpus sweep, and parity of these array semantics with generated code (the codegen side compares them separately). A string element is decoded into a std::string when it is read (src/array.cpp), and each transaction begin still makes one heap allocation for its handle until the allocation-free Store::begin() change lands.

Generated with Claude Code

luisleo526 and others added 10 commits October 27, 2026 14:04
…ber of bars

Add the array container (include/pineforge/array.hpp, src/array.cpp). General
arrays live in fixed pages over o1::Store with stable ids, and every public
call runs in one transaction, so a calc-on-fill rollback restores an array
byte for byte. A prepared path allocates fixed-size arrays before bar 0 so a
bar does no heap allocation. Pine semantics follow the vector path used by
generated code today: tolerant negative index reads give na, the checked v5 and
v6 index rules, value copies, nested arrays, drawing-handle arrays, and the
lower-timeframe clear and append order. There is no capacity refusal and no
dropped element.

Tests, registered only in tests/cr150.d/p2cx-a5arr.cmake: growth at 100,000
and 1,000,000 bars against a bound derived from the page size, header size and
the store's shape caps (the 1,000,000 row is a pinned slow row in
scripts/ci_verify.py), rollback, nested and handle arrays, preparation,
semantics and source hooks.

Known limits, stated in the sources: a string element is decoded into a
std::string when read (src/array.cpp), and each transaction begin still makes
one heap allocation for its handle (o1::Store contract).
…l file is unlinked

ScriptArray::operator=(ScriptArray&&) cleared the source even when the source
was a borrowed alias, so `x = y.alias()` emptied y. Only an owner is consumed
by a move now; a borrowed alias is copied from and left alone.
tests/test_array_semantics.cpp checks that the aliased array keeps its size and
values.

The array test fixture opened its spill file inside the source tree and never
removed it. It now opens a mkstemp file under TMPDIR (or /tmp) and unlinks it
right after opening, so a run leaves nothing behind and a read-only checkout
works.
…n every path

ScriptArrayTransaction::rollback() kept its handle when the store's rollback
threw, so the destructor rolled the same scope back a second time, and the
destructor turned a failed rollback into std::terminate although ~StoreTx
already abandons an unfinished handle without throwing. rollback() now moves
the handle out before acting, exactly like commit(), and the destructor body is
gone: the unique_ptr destroys the handle and the store abandons it noexcept.

tests/test_array_rollback.cpp gets three rows: an outer scope abandoned
uncommitted restores the array; an abandoned joined scope makes the outer
commit fail as rollback-only and restores the array; a rollback that fails in
the backing leaves the wrapper inactive and does not roll back again when it is
destroyed (the fixture backing can fail one write on request).

Test banners use the word "array" and the growth fixture identity is
"array-growth".
…line, count the new slow row

Verification of the array branch found three things that are not about array
behaviour:

- src/array.cpp is a kernel unit and called the codegen-contract helper
  pine_array_stop, which put that symbol into libpineforge_kernel.a and made
  check_kernel_residuals fail (the kernel boundary keeps the helpers out of the
  archive). The failures are now raised through a local function that throws the
  same coded pine_array_error with the same arguments and text, the way the
  other kernel units (for example matrix) do.
- Folding the kernel-sources comment onto one line moved `set(PINEFORGE_KERNEL_SOURCES`
  from line 118 to 117 and broke three documentation anchors. The two-line
  comment is back and src/array.cpp sits on the set() line, so line numbers are
  unchanged.
- The pinned slow-row count asserted by scripts/test_ci_verify.py is 61 with the
  new test_array_growth_1m row.
…e-bounds gate

Eleven new entries, all unreviewed, cover the thirty-four sites of the array
header and source: the numeric index parameters, the per-call decoded string
results, the Store reservation, the cold error-path strings and the small error
argument vector of the local array_stop helper. None claims a bound. The
inventory gains the matching rows, the unreviewed ceiling rises from 1264 to 1275
and the pinned open count from 1452 to 1463.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526
luisleo526 marked this pull request as ready for review October 27, 2026 14:02
@luisleo526
luisleo526 merged commit 4319d52 into main Oct 27, 2026
18 of 19 checks passed
luisleo526 added a commit that referenced this pull request Oct 27, 2026
…it (resident, disk, jailed arena) (#556)

memory: state the report-memory bound exactly and add the proofs for it (resident, disk, jailed arena)

The memory-management page said the default sink route keeps the equity
curve out of memory, but not what still grows with the bar count, and that
a jailed run without --report-sink-fd has no sink. It now states, with
sources: the resident equity-report buffers are bounded on the default
unjailed sink route; the curve scratch costs 24 B per script bar on disk
(6 MiB at 262,144 and 48 MiB at 2,097,152 bars) with the E output and the
JSON artifact; in a jailed run the scratch is the RAM-backed arena, so the
same 24 B per script bar are memory there, sized by the hosting wrapper and
failing closed with arena_exhausted; sink-off callers keep the native curve
plus an owning report copy. "O(1) in bars" is not claimed for the jailed
arena or the sink-off path.

Removed text, named. The page text below was introduced by 6d2bf95 (docs:
add the memory-management page, #531) and 30e9f9a (docs: memory-management
page re-read against main after the 1.8.0 changes, #546):
- the section "Use the report sink for bounded memory" is replaced by
  "Report memory: resident buffers, disk and the jailed arena" (its line
  citations were re-read and corrected);
- the phrase "a bounded default route is planned, with no release date" and
  the sentence "Install a report sink; no release date for the default
  route" are removed from the report-memory table (the page documents the
  route as it is);
- the two sink rows of that table are reworded to separate resident buffers
  from disk and arena growth; the sentences about the heap bound of the
  equity curve and the ledger, the sink file growth, and the legacy FULL sink
  route are rewritten into the new section, and the release container's
  default profile sentence of 30e9f9a is replaced by the route table.

Tests: a two-size resident proof for the report scratch curve (N = 262,144
and 8N script bars, RSS growth within 1 MiB, scratch growth exactly 24 B per
added bar, metric block bit-equal to the sink-off result); a corpus
projection test that the temporary-sink route and the sink-off route give
bit-identical metrics and diagnostics on real modules (it replaces the
named-sink summary comparison on corpus modules that this test did before,
and still skips when its manifest variable is unset). Both report-memory
rows are pinned as slow rows (61 pinned names) and registered before the
CR-150 include block. The binary-route peak-RSS row is left out of this
change: its 9-minute wall exceeds the CI leg budget.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

ci: count the array-growth slow row with the report-memory slow row (62 pinned names)

Both the report-memory change and the paged array store added one pinned slow
row each on top of 60, and git merged the two identical edits of the count
into 61. The pinned set now holds 62 names: 43 from the PINEFORGE_PR_SLOW_TESTS
loop and 19 direct CMake LABELS assignments.

The test assertion on len(PINNED_SLOW_TEST_ROWS) and the four sentences in
docs/ci.md that state the count and its split (introduced by 3ea5a0d, ci:
split the long test rows, shard the full CI legs and run them on the larger
hosted runners, #544, and rewritten by the report-memory change) are updated
to 62 and 43 + 19. No other text is removed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

ci: update the slow-row count literals in test_ci_preflight to 62, 43 and 19

test_documented_slow_row_counts_match_literal_population pins the three
numbers that docs/ci.md states for the pinned slow rows (all, from the
PINEFORGE_PR_SLOW_TESTS loop, direct LABELS assignments) and two sentences that
repeat them. The previous commit moved the documented count from 61 to 62
(43 + 19) but left this test at 61, 43 and 18, so test_ci_preflight_slice_6
failed on every profile. The test now states 62, 43 and 19. Only those literals
change.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

memory: move three docker/run_json.py citations after the bulk equity copy

The bulk path-sink equity copy added code above the report-sink helpers in
docker/run_json.py, so the three line ranges the memory-management page cites
for _ReportSink, _build_sink_report and _prepare_stream_report no longer
contained their symbols (doc-anchors: SYMMISS). The ranges now read 6183-6220,
7102-7148 and 6983-7003. Citation numbers only: no sentence is added, removed
or reworded.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>


Also rewritten: the 60-name slow-row counts in docs/ci.md introduced by 3ea5a0d (ci: split the long test rows ..., #544), now 62 = 43 + 19 (the report-memory row and #547's array-growth row); the Pine product host row of the report-memory table in docs/pages/memory-management.md introduced by 27cd77b (engine 1.8.0: cohort tables ..., #541), reworded to separate resident buffers from disk and arena growth.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant