Skip to content

Commit 4319d52

Browse files
luisleo526claude
andauthored
engine: Pine arrays stored on the paged object layer, flat in the number of bars (#547)
* engine: Pine arrays stored on the paged object layer, flat in the number of bars Add the array container (include/pineforge/array.hpp, src/array.cpp). General arrays live in fixed pages over o1::Store with stable ids, and every public call runs in one transaction, so a calc-on-fill rollback restores an array byte for byte. A prepared path allocates fixed-size arrays before bar 0 so a bar does no heap allocation. Pine semantics follow the vector path used by generated code today: tolerant negative index reads give na, the checked v5 and v6 index rules, value copies, nested arrays, drawing-handle arrays, and the lower-timeframe clear and append order. There is no capacity refusal and no dropped element. Tests, registered only in tests/cr150.d/p2cx-a5arr.cmake: growth at 100,000 and 1,000,000 bars against a bound derived from the page size, header size and the store's shape caps (the 1,000,000 row is a pinned slow row in scripts/ci_verify.py), rollback, nested and handle arrays, preparation, semantics and source hooks. Known limits, stated in the sources: a string element is decoded into a std::string when read (src/array.cpp), and each transaction begin still makes one heap allocation for its handle (o1::Store contract). * engine arrays: a moved-from borrowed alias keeps its array; test spill file is unlinked ScriptArray::operator=(ScriptArray&&) cleared the source even when the source was a borrowed alias, so `x = y.alias()` emptied y. Only an owner is consumed by a move now; a borrowed alias is copied from and left alone. tests/test_array_semantics.cpp checks that the aliased array keeps its size and values. The array test fixture opened its spill file inside the source tree and never removed it. It now opens a mkstemp file under TMPDIR (or /tmp) and unlinks it right after opening, so a run leaves nothing behind and a read-only checkout works. * engine arrays: ScriptArrayTransaction hands its handle to the store on every path ScriptArrayTransaction::rollback() kept its handle when the store's rollback threw, so the destructor rolled the same scope back a second time, and the destructor turned a failed rollback into std::terminate although ~StoreTx already abandons an unfinished handle without throwing. rollback() now moves the handle out before acting, exactly like commit(), and the destructor body is gone: the unique_ptr destroys the handle and the store abandons it noexcept. tests/test_array_rollback.cpp gets three rows: an outer scope abandoned uncommitted restores the array; an abandoned joined scope makes the outer commit fail as rollback-only and restores the array; a rollback that fails in the backing leaves the wrapper inactive and does not roll back again when it is destroyed (the fixture backing can fail one write on request). Test banners use the word "array" and the growth fixture identity is "array-growth". * engine arrays: raise array failures directly, keep the kernel source line, count the new slow row Verification of the array branch found three things that are not about array behaviour: - src/array.cpp is a kernel unit and called the codegen-contract helper pine_array_stop, which put that symbol into libpineforge_kernel.a and made check_kernel_residuals fail (the kernel boundary keeps the helpers out of the archive). The failures are now raised through a local function that throws the same coded pine_array_error with the same arguments and text, the way the other kernel units (for example matrix) do. - Folding the kernel-sources comment onto one line moved `set(PINEFORGE_KERNEL_SOURCES` from line 118 to 117 and broke three documentation anchors. The two-line comment is back and src/array.cpp sits on the set() line, so line numbers are unchanged. - The pinned slow-row count asserted by scripts/test_ci_verify.py is 61 with the new test_array_growth_1m row. * ci: record the allocation sites of the Pine array module in the source-bounds gate Eleven new entries, all unreviewed, cover the thirty-four sites of the array header and source: the numeric index parameters, the per-call decoded string results, the Store reservation, the cold error-path strings and the small error argument vector of the local array_stop helper. None claims a bound. The inventory gains the matching rows, the unreviewed ceiling rises from 1264 to 1275 and the pinned open count from 1452 to 1463. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
1 parent 8154993 commit 4319d52

18 files changed

Lines changed: 2435 additions & 4 deletions

‎CMakeLists.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ set(PINEFORGE_SOURCE_LAYER_SOURCES
115115
# The kernel: the independent backtest + forward-execution state machine.
116116
# These TUs never reach the source layer, so they also stand alone as
117117
# PineForge::kernel.
118-
set(PINEFORGE_KERNEL_SOURCES
118+
set(PINEFORGE_KERNEL_SOURCES src/array.cpp
119119
src/c_abi.cpp
120120
src/run_failure.cpp
121121
src/native_c_host.cpp

‎ci/cr150-source-bounds.yaml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ scan:
7474
generated_files:
7575
- "tutorial/**/generated*.cpp"
7676
ceilings:
77-
unreviewed: 1264
77+
unreviewed: 1275
7878
pending-flat-memory: 188
7979
owners:
8080
- {name: "generated-cpp", globs: ["tutorial/**"]}
@@ -1542,3 +1542,14 @@ entries:
15421542
- {id: "U1499", file: "src/native_order.cpp", symbol: "held_", ops: ["assign"], container: "unresolved by the index: held_ = ...", owner: "native-kernel", phase: "unclassified", state: "unreviewed", line: 2600}
15431543
- {id: "U1500", file: "src/source/pine_adapter.cpp", symbol: "cohorts", ops: ["decl", "grow"], container: "std::vector<const CohortRow*>", owner: "source-layer", phase: "unclassified", state: "unreviewed", line: 8761}
15441544
- {id: "U1501", file: "src/ta_misc.cpp", symbol: "held_", ops: ["assign"], container: "unresolved by the index: held_ = ...", owner: "ta-series", phase: "unclassified", state: "unreviewed", line: 602}
1545+
- {id: "U1511", file: "include/pineforge/array.hpp", symbol: "load_string", ops: ["decl"], container: "unresolved by the index: std::string", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: per-call string result. Site :67 declares the string bridge implemented at src/array.cpp:80,84. Per-call decoded element / string result, O(one element's bytes) per call, never retained by the array, NOT a per-bar growth, but UNBOUNDED in element length (closed by the later script-string work). Caller owns the returned string; retained array bytes live in Store. OPEN; ceiling +1 for this group. Bound named (P2-CX 17:20): none for element length. Reason unreviewed: the string bridge materializes the whole element and the gate has no state for a per-call result unbounded in length.", line: 67}
1546+
- {id: "U1512", file: "include/pineforge/array.hpp", symbol: "index", ops: ["assign", "decl"], container: "unresolved by the index: index = ... / Index", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: fixed-width integers / numeric index parameters, no growth. Site :86 is the normalized int64 index in checked; Index parameters at :422,431,449,455,461,467,471,475,506,513 are scalar numeric arguments, not containers (Index is an alias spelling recognized by the index elsewhere in the scanned tree). No allocation or retained growth. OPEN; ceiling +1 for this group of eleven sites. Bound named (P2-CX 17:20): fixed-width integer parameters, sizeof fixed, no container. Reason unreviewed: the gate bounded form covers only guarded scalar vectors and refuses entries under include/.", line: 86}
1547+
- {id: "U1513", file: "include/pineforge/array.hpp", symbol: "value", ops: ["assign", "decl"], container: "unresolved by the index: Index / value = ...", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: scalar index parameter and per-call decoded element. Site :94 is an Index numeric parameter (fixed-width integers/scalars, no growth); :269 remove_at and :482 mutate decode one T. Scalar/handle values are fixed cells; nested arrays are fixed handles over cloned Store bytes. For string T: per-call decoded element / string result, O(one element's bytes) per call, never retained as array-resident C++ storage, NOT a per-bar growth, but UNBOUNDED in element length (closed by the later script-string work). Removal returns the value to its caller; mutation re-encodes it into Store. OPEN; ceiling +1 for this group of three sites. Bound named (P2-CX 17:20): scalar and handle elements are fixed cells (8 B double or int64, 4 B drawing handle, 1 B bool); a string element is the same unbounded-length gap as the string bridge entry U1511. Reason unreviewed: the gate bounded form does not apply under include/.", line: 94}
1548+
- {id: "U1514", file: "include/pineforge/array.hpp", symbol: "decode", ops: ["decl"], container: "unresolved by the index: std::string", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: per-call decoded string result. Site :144 ElementCodec<std::string, void>::decode delegates to load_string. Per-call decoded element / string result, O(one element's bytes) per call, never retained by the array, NOT a per-bar growth, but UNBOUNDED in element length (closed by the later script-string work). Caller owns the returned string. OPEN; ceiling +1 for this group. Bound named (P2-CX 17:20): none for element length (delegates to the string bridge U1511). Reason unreviewed: unbounded per-call string result until the codegen path reads strings through ScriptString.", line: 144}
1549+
- {id: "U1515", file: "include/pineforge/array.hpp", symbol: "result", ops: ["grow"], container: "unresolved by the index: result.reserve(...) / result.push_back(...)", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: Store-backed result construction and per-call element bridge. Sites :295 clone_into and :362 preallocated reserve logical ScriptArray capacity; :368 from pushes each initializer element. These are ScriptArray calls, not resident vector/string growth: logical bytes spill through Store, whose object holds them; population-sized logical reservation has no resident-derived cap. Any per-call decoded element / string result is O(one element's bytes) per call, never retained as array-resident C++ storage, NOT a per-bar growth, but UNBOUNDED in element length (closed by the later script-string work). The returned array keeps Store bytes, not a C++ element directory; total logical result bytes can scale with element count. OPEN; ceiling +1 for this group of three sites. Bound named (P2-CX 17:20): the resident part is the fixed three-word array handle, and each element moves through the fixed 4,096-byte kPayloadPageBytes stack buffer; logical capacity lives in the paged Store. Reason unreviewed: logical capacity has no resident-derived cap and the gate bounded form does not apply under include/.", line: 295}
1550+
- {id: "U1516", file: "include/pineforge/array.hpp", symbol: "store_", ops: ["grow"], container: "unresolved by the index: store_.resize(...)", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: Store reservation (logical, spills; the Store object holds the bytes). Site :491 reserve calls Store::resize(id_, offset(next)); the id stays stable and logical capacity may grow without a resident-derived cap. Successful resize delegates to Store, with no array-resident element vector or directory. This is not a flat-memory proof. OPEN; ceiling +1 for this group. Bound named (P2-CX 17:20): logical growth of ONE Store object; resident pages are capped by kMaxResidentPages (1024 pages), the rest spills; no C++ heap. Reason unreviewed: the logical size is script data with no resident-derived cap, and the gate bounded form does not apply under include/.", line: 491}
1551+
- {id: "U1517", file: "src/array.cpp", symbol: "<temp:string>", ops: ["temp"], container: "unresolved by the index: std::string", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: cold error-path strings of fixed size, plus an empty max_size probe. Sites :12,19,20,41,55 construct fixed-vocabulary diagnostic strings; numeric spellings use fixed-width types and method names come from fixed literals. No retained diagnostics or per-bar growth. Site :83 is the empty std::string{} used for max_size() in load_string, a fixed-size temporary on the normal validation path, not the element materialization at :84. OPEN; ceiling +1 for this group of six sites. Bound named (P2-CX 17:20): fixed literal vocabulary plus one 64-bit decimal, at most about 120 bytes, thrown once and not retained. Reason unreviewed: the gate bounded form covers only guarded scalar vectors.", line: 12}
1552+
- {id: "U1518", file: "src/array.cpp", symbol: "std::to_string", ops: ["temp"], container: "unresolved by the index: std::to_string(...)", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: cold error-path strings of fixed size. Sites :13,14 index_error and :21,23 index_error_double format fixed-width int64/double indices and uint64 sizes; numeric text has a fixed maximum size independent of bars or element count. Locals are discarded after the error, never collected or retained. OPEN; ceiling +1 for this group of four sites. Bound named (P2-CX 17:20): one int64, uint64 or double formatted to decimal, at most about 32 bytes, discarded after the throw. Reason unreviewed: the gate bounded form covers only guarded scalar vectors.", line: 13}
1553+
- {id: "U1519", file: "src/array.cpp", symbol: "load_string", ops: ["decl"], container: "unresolved by the index: std::string", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: per-call string result. Site :80 defines load_string and returns the local at :84 to its caller. Per-call decoded element / string result, O(one element's bytes) per call, never retained by the array, NOT a per-bar growth, but UNBOUNDED in element length (closed by the later script-string work). Store::size(id) determines materialization length; std::string::max_size is not a resident-memory bound. OPEN; ceiling +1 for this group. Bound named (P2-CX 17:20): none for element length. Reason unreviewed: the string bridge materializes the whole element; closed when the codegen path reads strings through ScriptString.", line: 80}
1554+
- {id: "U1520", file: "src/array.cpp", symbol: "result", ops: ["decl"], container: "unresolved by the index: std::string", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A head 0e15fc206067), class: per-call decoded string materialization. Site :84 allocates count bytes from Store::size(id), fills the local string and returns it to the caller. Per-call decoded element / string result, O(one element's bytes) per call, never retained by the array, NOT a per-bar growth, but UNBOUNDED in element length (closed by the later script-string work). The later script-string work must close this remaining gap; no flat-memory proof is claimed. OPEN; ceiling +1 for this group. Bound named (P2-CX 17:20): none for element length. Reason unreviewed: the string bridge allocates the element size read from the Store; closed when the codegen path reads strings through ScriptString.", line: 84}
1555+
- {id: "U1521", file: "src/array.cpp", symbol: "args", ops: ["decl", "grow"], container: "unresolved by the index: RunFailureArgs", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "CR150-CX 10-27 (A5A follow-up), class: cold error-path vector in the [[noreturn]] array_stop helper. Site :16 constructs the reason key/value pair; site :17 optionally appends the method pair. At most two pairs, a fixed count independent of bars or element count. The local vector is discarded by the throw, never retained, and is not a per-bar path. No flat-memory proof is claimed. OPEN; ceiling +1 for this group of two sites. Bound named (P2-CX 17:20): at most 2 key/value pairs (reason, method), discarded by the throw. Reason unreviewed: the gate bounded form covers only guarded scalar vectors.", line: 16}

0 commit comments

Comments
 (0)