Repository navigation
engine: Pine arrays stored on the paged object layer, flat in the number of bars - #547
Merged
Merged
Conversation
…ber of bars Add the array container (include/pineforge/array.hpp, src/array.cpp). General arrays live in fixed pages over o1::Store with stable ids, and every public call runs in one transaction, so a calc-on-fill rollback restores an array byte for byte. A prepared path allocates fixed-size arrays before bar 0 so a bar does no heap allocation. Pine semantics follow the vector path used by generated code today: tolerant negative index reads give na, the checked v5 and v6 index rules, value copies, nested arrays, drawing-handle arrays, and the lower-timeframe clear and append order. There is no capacity refusal and no dropped element. Tests, registered only in tests/cr150.d/p2cx-a5arr.cmake: growth at 100,000 and 1,000,000 bars against a bound derived from the page size, header size and the store's shape caps (the 1,000,000 row is a pinned slow row in scripts/ci_verify.py), rollback, nested and handle arrays, preparation, semantics and source hooks. Known limits, stated in the sources: a string element is decoded into a std::string when read (src/array.cpp), and each transaction begin still makes one heap allocation for its handle (o1::Store contract).
…to the arrays branch
…l file is unlinked ScriptArray::operator=(ScriptArray&&) cleared the source even when the source was a borrowed alias, so `x = y.alias()` emptied y. Only an owner is consumed by a move now; a borrowed alias is copied from and left alone. tests/test_array_semantics.cpp checks that the aliased array keeps its size and values. The array test fixture opened its spill file inside the source tree and never removed it. It now opens a mkstemp file under TMPDIR (or /tmp) and unlinks it right after opening, so a run leaves nothing behind and a read-only checkout works.
…n every path ScriptArrayTransaction::rollback() kept its handle when the store's rollback threw, so the destructor rolled the same scope back a second time, and the destructor turned a failed rollback into std::terminate although ~StoreTx already abandons an unfinished handle without throwing. rollback() now moves the handle out before acting, exactly like commit(), and the destructor body is gone: the unique_ptr destroys the handle and the store abandons it noexcept. tests/test_array_rollback.cpp gets three rows: an outer scope abandoned uncommitted restores the array; an abandoned joined scope makes the outer commit fail as rollback-only and restores the array; a rollback that fails in the backing leaves the wrapper inactive and does not roll back again when it is destroyed (the fixture backing can fail one write on request). Test banners use the word "array" and the growth fixture identity is "array-growth".
…line, count the new slow row Verification of the array branch found three things that are not about array behaviour: - src/array.cpp is a kernel unit and called the codegen-contract helper pine_array_stop, which put that symbol into libpineforge_kernel.a and made check_kernel_residuals fail (the kernel boundary keeps the helpers out of the archive). The failures are now raised through a local function that throws the same coded pine_array_error with the same arguments and text, the way the other kernel units (for example matrix) do. - Folding the kernel-sources comment onto one line moved `set(PINEFORGE_KERNEL_SOURCES` from line 118 to 117 and broke three documentation anchors. The two-line comment is back and src/array.cpp sits on the set() line, so line numbers are unchanged. - The pinned slow-row count asserted by scripts/test_ci_verify.py is 61 with the new test_array_growth_1m row.
…e-bounds gate Eleven new entries, all unreviewed, cover the thirty-four sites of the array header and source: the numeric index parameters, the per-call decoded string results, the Store reservation, the cold error-path strings and the small error argument vector of the local array_stop helper. None claims a bound. The inventory gains the matching rows, the unreviewed ceiling rises from 1264 to 1275 and the pinned open count from 1452 to 1463. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526
marked this pull request as ready for review
October 27, 2026 14:02
luisleo526
added a commit
that referenced
this pull request
Oct 27, 2026
…it (resident, disk, jailed arena) (#556) memory: state the report-memory bound exactly and add the proofs for it (resident, disk, jailed arena) The memory-management page said the default sink route keeps the equity curve out of memory, but not what still grows with the bar count, and that a jailed run without --report-sink-fd has no sink. It now states, with sources: the resident equity-report buffers are bounded on the default unjailed sink route; the curve scratch costs 24 B per script bar on disk (6 MiB at 262,144 and 48 MiB at 2,097,152 bars) with the E output and the JSON artifact; in a jailed run the scratch is the RAM-backed arena, so the same 24 B per script bar are memory there, sized by the hosting wrapper and failing closed with arena_exhausted; sink-off callers keep the native curve plus an owning report copy. "O(1) in bars" is not claimed for the jailed arena or the sink-off path. Removed text, named. The page text below was introduced by 6d2bf95 (docs: add the memory-management page, #531) and 30e9f9a (docs: memory-management page re-read against main after the 1.8.0 changes, #546): - the section "Use the report sink for bounded memory" is replaced by "Report memory: resident buffers, disk and the jailed arena" (its line citations were re-read and corrected); - the phrase "a bounded default route is planned, with no release date" and the sentence "Install a report sink; no release date for the default route" are removed from the report-memory table (the page documents the route as it is); - the two sink rows of that table are reworded to separate resident buffers from disk and arena growth; the sentences about the heap bound of the equity curve and the ledger, the sink file growth, and the legacy FULL sink route are rewritten into the new section, and the release container's default profile sentence of 30e9f9a is replaced by the route table. Tests: a two-size resident proof for the report scratch curve (N = 262,144 and 8N script bars, RSS growth within 1 MiB, scratch growth exactly 24 B per added bar, metric block bit-equal to the sink-off result); a corpus projection test that the temporary-sink route and the sink-off route give bit-identical metrics and diagnostics on real modules (it replaces the named-sink summary comparison on corpus modules that this test did before, and still skips when its manifest variable is unset). Both report-memory rows are pinned as slow rows (61 pinned names) and registered before the CR-150 include block. The binary-route peak-RSS row is left out of this change: its 9-minute wall exceeds the CI leg budget. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> ci: count the array-growth slow row with the report-memory slow row (62 pinned names) Both the report-memory change and the paged array store added one pinned slow row each on top of 60, and git merged the two identical edits of the count into 61. The pinned set now holds 62 names: 43 from the PINEFORGE_PR_SLOW_TESTS loop and 19 direct CMake LABELS assignments. The test assertion on len(PINNED_SLOW_TEST_ROWS) and the four sentences in docs/ci.md that state the count and its split (introduced by 3ea5a0d, ci: split the long test rows, shard the full CI legs and run them on the larger hosted runners, #544, and rewritten by the report-memory change) are updated to 62 and 43 + 19. No other text is removed. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> ci: update the slow-row count literals in test_ci_preflight to 62, 43 and 19 test_documented_slow_row_counts_match_literal_population pins the three numbers that docs/ci.md states for the pinned slow rows (all, from the PINEFORGE_PR_SLOW_TESTS loop, direct LABELS assignments) and two sentences that repeat them. The previous commit moved the documented count from 61 to 62 (43 + 19) but left this test at 61, 43 and 18, so test_ci_preflight_slice_6 failed on every profile. The test now states 62, 43 and 19. Only those literals change. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> memory: move three docker/run_json.py citations after the bulk equity copy The bulk path-sink equity copy added code above the report-sink helpers in docker/run_json.py, so the three line ranges the memory-management page cites for _ReportSink, _build_sink_report and _prepare_stream_report no longer contained their symbols (doc-anchors: SYMMISS). The ranges now read 6183-6220, 7102-7148 and 6983-7003. Citation numbers only: no sentence is added, removed or reworded. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Also rewritten: the 60-name slow-row counts in docs/ci.md introduced by 3ea5a0d (ci: split the long test rows ..., #544), now 62 = 43 + 19 (the report-memory row and #547's array-growth row); the Pine product host row of the report-memory table in docs/pages/memory-management.md introduced by 27cd77b (engine 1.8.0: cohort tables ..., #541), reworded to separate resident buffers from disk and arena growth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the engine's Pine array container,
pineforge::ScriptArray<T>(include/pineforge/array.hpp,src/array.cpp). Arrays live in fixed pages overo1::Storewith stable ids, so the memory an array retains does not grow with the number of bars, and every public call runs in one transaction so a calc-on-fill rollback restores an array byte for byte.o1::Store, stable ids,Store::Txrollback.std::vectorpath the generated code uses today: tolerant negative-index reads givena, the checked v5 and v6 index rules, value copies, nested arrays, drawing-handle arrays and the lower-timeframe clear/append order. There is no capacity refusal and no dropped element.ScriptArrayTransactionhands its handle to the store on every path:rollback()moves the handle out before acting, likecommit(), and a failed rollback no longer rolls the scope back a second time or terminates the process.x = y.alias()copies from the borrowed alias and leavesyintact; only an owning array is consumed by a move.Tests
Registered in
tests/cr150.d/p2cx-a5arr.cmake; the 1,000,000-bar growth row is a pinned slow row inscripts/ci_verify.py.test_array_growth_100k,test_array_growth_1m)test_array_semantics,test_array_rollback,test_array_nested_handles,test_array_preallocated,test_array_source_hooks)Evidence from
lab remotectest-onlyruns (Release): on the final head, all seventest_array_*tests pass (rj-20261027t064709-c8fa54). Reverting the alias-move fix failstest_array_semanticswitharray size differs from vector. Reverting the transaction-wrapper fix failstest_array_rollbackwitharray rollback: throwing rollback retained a stale transaction handle(rj-20261027t064713-0c9fe7). The alias-move job on the earlier head wasrj-20261027t050111-3f307a.Not covered
Sanitizers, the Debug and kernel profiles, macOS, the corpus sweep, and parity of these array semantics with generated code (the codegen side compares them separately). A string element is decoded into a
std::stringwhen it is read (src/array.cpp), and each transaction begin still makes one heap allocation for its handle until the allocation-freeStore::begin()change lands.Generated with Claude Code