Skip to content

engine: fixed-shape O(1) store for Pine collections (A5-STORE) - #515

Merged
luisleo526 merged 14 commits into
mainfrom
p2/a5s2
Oct 25, 2026
Merged

luisleo526 merged 14 commits into
mainfrom
p2/a5s2

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

A5-STORE (1.7.1 plan section B, P2-CX): a fixed-shape O(1) chunked store for Pine collections (include/pineforge/detail/o1_store.hpp, src/detail/o1_store.cpp): chunk buffer and resident pages capped in the constructor (kMaxChunkBytes 64 KiB, kMaxResidentPages 1024) with a typed refusal at the cap; transactions with rollback; nothing in the engine uses it yet.

Tests: tests/test_o1_store.cpp (chunk boundary, id monotonicity, resident ceiling, rollback restore, shape caps) with five mutations each turning a REQUIRE red; lab ctest green. The CR-150/C4 source allocation gate gets reviewed entries for its six new sites (CR150-CX), ceiling changes named per site.

Not proven here: no kernel/release/debug/sanitizer run of this exact head beyond PR CI; macOS not run (ci-post). Engine merge gate: needs verify + parity (lab) on this head.

🤖 Generated with Claude Code

luisleo526 and others added 14 commits October 25, 2026 23:51
Add include/pineforge/detail/o1_store.hpp (stable-id objects, in-place
partial writes, resize, byte-exact transaction rollback; interface as the
CR-150 A5 prior art) and src/detail/o1_store.cpp, which implements it as a
thin layer over ScratchStore: id = monotonic counter reserved before
staging, object = header record + fixed-size chunk records, Tx maps onto
ScratchStore begin/commit/rollback. One kernel source line in CMakeLists.txt.

tests/test_o1_store.cpp (public API only) covers lifecycle, resize, chunk
boundaries, a multi-chunk rollback, monotonic ids after a failed
allocation, the 100k/1M-bar resident ceiling against a std::vector RED
control, and no heap allocation after open. Registered in
tests/cr150.d/p2cx-a5s.cmake (slow;mem).

Edit-only lane: nothing built or run at commit time; lab verification follows.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
create() wrote the 16-byte header at the head of the shared chunk buffer and
then reused the same buffer for chunks 1..n, so every data chunk began with the
header bytes instead of zeros. Zero the buffer before the data-chunk loop.
Found by test_o1_store (chunk-boundary read at offset 15, lab row
rj-20261024t154037-8f329c).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The committed-transaction group of check_no_heap_after_open failed on the lab
(rj-20261024t155803-5531ae, REQUIRE(held) at the group end) without saying
which step. Print the step reached and any exception text so the next lab row
names the cause.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The discarded-transaction group created a 48-byte object and expected a
64-byte write to return 64; Store::write clips to the object size. Expect 48.
Lab row rj-20261024t160559-716c38 showed the committed group now passes and
this group fails.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The discarded-transaction group created the object before begin(), so the
rollback could not discard it and `!alive(discarded)` was false. Create it
inside the transaction. Lab row rj-20261024t161347-e68974 (REQUIRE(held) at
the end of the group, the committed group before it passes).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
….txt keeps its line count

An inserted line shifts every later line of CMakeLists.txt and breaks seven doc anchors
(CMakeLists.txt:169, :172, :398 cited by native-engine.md, adr/0001, native-feature-parity.md);
joining the new source onto an existing list line keeps the anchors valid with no doc edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ges 1024), refused in the constructor

The chunk buffer and the resident page cache were sized straight from Shape, with only a size_t
overflow guard above. The CR-150/C4 source-bound gate can name a fixed input only if the constructor
itself refuses larger values, so the two limits (values set by CR150-CX) are now named constants and
std::invalid_argument refusals next to the existing Shape checks. New test group check_shape_caps:
a Shape at each cap opens (chunk cap also round-trips a full-size object), one above each cap is refused.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…d onto add_library so line numbers stay put

One kernel source per line is what scripts/measure_detached_comments.py kernel_closure() reads, so the
earlier same-line join broke the detached-comments stage. Listing the source on its own line shifts
the lines below it by one; moving the explanatory comment above add_library to the end of that
line gives the line back. CMakeLists.txt stays at its line count, add_library stays on line 169
and PINEFORGE_KERNEL_SOURCES on 118, so the doc anchors that cite them still hold.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…hored by CR150-CX), ceiling 1237 -> 1241

Applies CR150-CX's gate patch (runner/CR150-CX/gate-patch-p2-a5s2.patch, sha256 a825b090efac2ff6...):
ci/cr150-source-bounds.yaml gains rows U1483-U1486 (state unreviewed, each note names its site and
class, the Shape caps kMaxChunkBytes / kMaxResidentPages are named in the reset group's note),
ceilings.unreviewed 1237 -> 1241; ci/cr150-source-sites.json gains the six site records copied from the
gate's own findings; scripts/test_cr150_source_gate.py pins the OPEN count 1425 -> 1429.
Data authored by CR150-CX, not by the a5s lane; no gate script logic changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…gin() allocate before it opens; tests that can see a per-object leak

REV-CX's two P2 findings on the A5 object layer.
P2-2: lookup/read failures (header lookup/read, chunk lookup/read) threw through throw_status
without marking the open outer transaction rollback-only, so a caller that caught the exception
after an earlier chunk had been written could commit a torn write. throw_status is now a StoreImpl
member and marks rollback_only_ before it builds the exception string. begin() also allocates its
handle BEFORE it opens the ScratchStore transaction, so a bad_alloc cannot leave an active outer
transaction nobody owns. New rows: a read-fault backing that fails the second chunk's value read
after the first chunk's put (write throws, the next mutation throws logic_error, rollback restores
the object byte-exactly) and a refused begin allocation.
P2-1: the growth-gate "resident flat" assertions read ScratchStore's constant workspace size and
cannot see an object-layer heap leak; the growth loop now also asserts that the number of global
allocations equals one Tx handle per begin() at both 100,000 and 1,000,000 bars.
P3: zero-fill assertion after shrink and regrow, contract text (begin() throws while rollback-only,
handles must be destroyed before the next outer begin, single-threaded, one allocation per begin),
the resident-page cap comment (a page-count cap, not a byte cap), the test ceiling's metadata term,
the journal_slots comment, and the debug fprintf lines are gone.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… StoreTx site, throw_status in the class); header contract clause

Applies CR150-CX's re-authored gate data (runner/CR150-CX/gate-patch-p2-a5s2-76fceda8ad.patch, sha256
eb3f0a9fa1f326b4bdb839d31a5e06d7629ff03078b28f67a07a58566a350b64): the <temp:string> record takes the
scope of throw_status as a StoreImpl member, the two new:StoreTx records become one record with the
symbol `transaction` (begin() now allocates its handle once, before the ScratchStore transaction
opens), the notes carry the current lines and the page-COUNT wording of the resident cap (resident =
(resident_pages + 6 work pages + metadata pages) x page_bytes, at most about 1 GiB at ScratchStore's
1 MiB page limit), and the OPEN pin follows the gate's own count (1429 -> 1428). Data authored by
CR150-CX; no gate script logic changes.

o1_store.hpp: the contract now says that a backing failure in any call, including a read-only one,
makes the open outer transaction rollback-only (REV-CX P3). Comment only; the cap lines 54-55 and
every o1_store.cpp line stay where they were.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The gate on 3b6aa07 reports containers=1429 entries=1429 findings=0 (OPEN counts entries, not
sites; the four o1_store rows all match again). The previous commit pinned 1428 from the unmatched
state of one row; scripts/test_cr150_source_gate.py:1218 returns to the value it had before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
tests/cr150.d/p2cx-a5s.cmake:3 labels test_o1_store "slow;mem" (the test runs about five minutes). The
sharded Debug profile excludes slow rows and refuses any excluded row that is not in
PINNED_SLOW_TEST_ROWS, so the Debug verify leg failed at the ctest-exclusion stage with "CTest excluded
unpinned slow rows: test_o1_store" (lab job rj-20261025t173910-db2c79, release leg green). The policy
comment asks for an explicit change for a new slow row: this adds the one name, after the last existing
entry, so no earlier line of scripts/ci_verify.py moves.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in 6cfea80 (docs: backfill release notes for engine releases, #510): CHANGELOG.md and
README.md only. First parent is 52cd416; no file of the o1::Store change is touched by main.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526
luisleo526 merged commit 180630f into main Oct 25, 2026
13 checks passed
luisleo526 added a commit that referenced this pull request Oct 26, 2026
## What

Adds the `## 1.8.0 — 2026-10-27` section (with migration notes) to `CHANGELOG.md`, above `## 1.7.0`. `CHANGELOG.md` is the only file changed (+131/-0). Documentation only: no `src/`, `include/`, `docker/`, `scripts/`, `tests/` or workflow file is touched.

## Why

The engine release body is generated from git-log subjects only and carries no migration notes; the release notes users read are this file (and the GitHub release body edited from it after publish). The notes must be on main before the release dispatch.

## Content

- Compatibility and migration: exact pairing with codegen 1.8.0, regenerate C++ and relink every strategy library, no public header change, the new codegen `color` input type, the changed harness file `docker/run_json.py`.
- Faster `run_json` report handling (#517) with the measured table against 1.6.3 and 1.7.0 and the statement that the gap to 1.6.3 is not closed.
- Scratch placement restated (#518); the unused storage building block (#515; its allocation-gate sites are unreviewed, no bound claimed); a known limit (memory with never-filled pending orders, "a fix is planned for a later release").
- CI and release process: every merged change since 1.7.0 not listed above (#505-#510, #513, #514, #516, #519, #520, #522, #469, #481, #504). Every line is taken from the merged commit body; none claims a behaviour change for a tests-only or CI-only change.

Not in this commit (not on main at base `641349a`): the release gate, tmpwatch and the TvJO fix. Each gets one line from its merged PR body as a follow-up commit on this branch before this PR merges; this PR is meant to merge after those three.

## Checks

- Route: docs-only, `lab verify --via ci-only` (no sweep).
- Read as a committed diff by an independent reviewer: GO. All 18 engine commits `v1.7.0..641349a` have a bullet, none extra.
- Numbers: the #517 timings and RSS are the measurements in the #517 body; the ratios in the text were recomputed (1.77/0.36 = 4.9, 8.26/1.35 = 6.1, 8.94/5.39 = 1.66; 4.61/0.36 = 12.8, 22.31/1.35 = 16.5, 22.99/5.39 = 4.27).
- The heading date `2026-10-27` is the planned tag date and is corrected before the tag if the date moves.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant