Skip to content

Memory caches, equity-curve window, cohort exposure fold and admission batches - #679

Merged
luisleo526 merged 41 commits into
mainfrom
train/t4
Oct 31, 2026
Merged

luisleo526 merged 41 commits into
mainfrom
train/t4

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

One merge train of four reviewed changes, built on main after #675:

  • Memory caches (Memory caches: one shared read-through cache, a versioned per-kind cache policy, an allocation-failure bypass, and cache diagnostics for hosted callers #670): one shared read-through cache with a versioned per-kind cache policy and an allocation-failure bypass; the parent terminal scratch test goes back to main's journal refusal row.
  • Equity-curve window: a sequential pass reads whole blocks again (the curve read cost and the jailed projection are pinned), the block reset before a block read is pinned, and two documentation fixes.
  • Cohort exposure fold: the overflow path no longer walks the whole roster; the completion threshold is pinned to the retained roster size.
  • Admission journal fold and exit scratch batches: a new event folds in one validated, hashed pass; live-handle copies publish in batches of 64 rows when no I/O observer is installed; the two rethrows are admitted in the run-failure throw allowlist.

Main is merged once (its tree equals the head this train was built on, so the merge keeps the train tree). Each member had an independent review before joining.

🤖 Generated with Claude Code

luisleo526 and others added 30 commits October 30, 2026 17:08
Move the exposure fold of a cohort out of PineExecutionAdapter into a free
function over the origin roster and the live-unit map, appended at the end of
the adapter translation unit. The old selector moves verbatim: same visit
order, same additions, same final clamp, so results are bit-identical.

The member overload keeps its exact source footprint, so later line citations
do not move. A values-only property test compares the seam with an independent
frozen copy of the old selector byte for byte over magnitude mixes, repeated
incarnations, holes, retired prefixes, overflow and special doubles.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Report the work of every exposure fold through the existing work macro (calls,
unit-row visits, retained window, live rows) from the member wrapper, and add
rows that pin the growth: on one reused entry id the fold is cubic in the
number of openings, on distinct ids it is quadratic. The rows gate the doubling
ratio and absolute bounds, so they fail on the current selector and pass for
any fold that is linear per call.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Replace the quadratic position selector by an exact fold: walk the retained
roster window once when it is small against the live rows, otherwise gather the
(position, units) pairs into a fixed stack buffer, sort them by position and
fold in that order. Both paths add the same doubles in the same order with the
same multiplicity and filter as before, so every result is bit-identical; an
empty roster or an empty live-unit map returns zero early. No heap, no cache,
no new member: the fold uses only the existing roster and map.

An opt-in fixture build compares every call with a frozen copy of the old
selector and aborts on any difference in the bytes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…p row

Add rows on a roster forced onto the sorted path: NaN, infinities, negative
zero, zero, a negative value and the smallest denormal mixed with positive
rows, a maximum-double overflow that must clamp to zero, and interleavings whose
ascending sum differs from every grouped order, all compared bit for bit with
the frozen selector. Remove the caller-group registration from the product test
list (it only skipped without its observation overlay) and extend the oracle
option to the two quiet-bar libraries that compile the adapter.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ke tail intact

Register the two exposure fold test executables from their own lane file instead
of inserting them into the middle of the main test list, and place the opt-in
oracle option immediately before the lane include block, so no earlier line of
the main CMake file moves and the fast-row registration block stays the last
CMake code.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…nventory

The exposure fold moved out of the adapter class into a pair of free functions at
the end of the adapter source. The reviewed site inventory still held the three
assign records of the removed member function, so the source-bounds gate reported
two stale identities and five new ones, and three tests that run the gate on the
real tree failed with it.

Replace the three removed records with the five records the gate names: the
position, remaining and next assignments of the fold and the position and next
assignments of its checker, each with the scope text and ordinal the gate printed.
No manifest entry changes state and no entry is added or removed, so the entry
and unreviewed counts stay as they are.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…r its encoding

The admission journal fold used to scan an event record to validate it and then read the same record a second time to hash it, so every appended event cost two reads of each of its pages (16 source reads for a 3,773-byte record on 512-byte pages). A new fused fold, fold_validated_journal_record, validates and hashes the same forward field stream in a single pass (8 reads for that record) on both the bound and the unbound path. The standalone scanner and folder remain for the other consumers.

The result is identical: digest, summary, sequence, rejection gate and the published frontier are unchanged, and a failure publishes the same status and diagnostic as before. After a long-text skip the old scanner's refill alignment differs from the fused reader's, so on a source read failure the fused fold replays the validation once to recover the old error priority; it never retries the hash and never turns the original read failure into success.

The workspace is a fixed constant: a compile-time ledger keeps the fold's own state, diagnostic transport, summaries and controls within 1,024 bytes, with a 256-byte reused field buffer and no new heap allocation. No state survives the call.

Tests: the fold-cost row now counts work (reads, encoder words and writer-status checks per new event, derived from the schema's field geometry and the page size, with the unchanged window costing zero), a failure-publication row corrupts a stored record and checks that nothing is published, and a new differential unit compares the fused fold with the separate composition for all event kinds, text and array sizes, page sizes, malformed encodings and byte-range I/O faults. The schema checker and its mutation control name the new call.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…en no I/O observer is installed

The adapter's two flat live-handle copies (the unconditional snapshots taken when an Applied callback finds a flat position) put each handle into the exit scratch as its own transaction, so N rows cost N publications. They now run in one insertion transaction that publishes every 64 rows (the existing exit-scope batch constant) and once more for a nonempty tail: 0, 1, 64 and 65 rows publish 0, 1, 1 and 2 times instead of 0, 1, 64 and 65. The transaction is finished before the consumer loop starts, so no builder is open at Applied, consumer, hash, callback or teardown boundaries.

The batched form is eligible only when the bound backing carries no I/O observer and the context has no row or lookup observer. Production installs no observer (the one source-side registration clears it for derived identities and restores it); observers exist in measurement and test harnesses, which therefore see exactly the charge sequence they saw before. ScratchBacking and CommandScratchStore gain a const has_io_observer query that reads the already-present observer member, and the observer's comment now states its contract: a passive accounting sink that must not call back into the engine, a store or a consumer. All other producers keep the per-row path.

No state is added: the mode is a template parameter and the store's existing fixed transaction workspace does the buffering.

Tests: a construction-publication row counts work (puts, fresh handles, publications) for 0, 1, 64 and 65 rows with no observer, and an observed row requires N publications and the same charge-by-charge sequence as the per-row reference. A test comment now states that 64 is a batch constant, not a page capacity (an ordered leaf at 512-byte pages holds 58 keys).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…che policy, an allocation-failure bypass, and cache diagnostics for hosted callers

The engine's resident caches (command scratch pages, id-table pages, the trade ledger's decoded rows, feed projections, TA history windows, tail/sequence/curve windows, and the native journal's pages) now share one read-through cache component and one policy. Every cache is a fixed constant chosen at run begin, independent of bar and trade counts; a read that misses goes to the scratch backing, and an optional cache that cannot allocate is bypassed instead of failing the run.

## Cache policy

- A new additive C export, strategy_set_cache_policy_v1 with pf_cache_policy_v1 / pf_cache_policy_entry_v1, sets a per-kind resident byte allowance. Omitted kinds keep their defaults; NULL restores them; duplicate or unknown kinds, nonzero reserved words and an unrepresentable size are typed setting_rejected errors. A run freezes its settings before its first store begin or append. PF_ABI_VERSION and every existing layout are unchanged.
- An allowance is inclusive: it covers each line's metadata, not only its payload. A capacity below one line bypasses caching for that kind; reads still use the backing.
- The trade ledger keeps its private pages (256 slots per page, lazy allocation, growth that preserves entries, release outside the retained window). Its allowance selects the largest row count R with R * 312 + ceil(R / 256) * sizeof(void*) <= capacity_bytes, converted once; the default is 65,536 rows (20,449,280 bytes on LP64) with or without a report sink. A nonzero capacity converting to fewer than one row, or above the 1,048,576-row ceiling, is a typed setting error.
- The native journal now records events of any size across as many pages as they need (the former one-page refusal is gone); the TA history cache, the feed projections, the sequence index and the tail windows account for their line metadata inside their allowances.

## Allocation failure is a bypass, not a failure

An optional cache allocation that fails (the command cache's slab, the ledger's directory or a page) permanently bypasses further cache allocation for that owner and the run continues through the backing, with a typed count. Required allocations keep the typed out_of_memory; a syscall error keeps scratch_io_failed.

## Diagnostics for hosted callers

- A new additive export, pf_report_cache_stats, reads the run's resident cache bytes (each allocation owner counted once) and its allocation-bypass count after report finalization.
- docker/run_json.py adds cache_resident_bytes, cache_bypass_reason ("alloc_failed") and allocation_bypass_count to the report diagnostics only when a bypass occurred; a run without a bypass has the same report bytes as before.

## Tests, gates and documentation

Cache sweep rows for every kind, allocation-failure and fault-injection rows, lazy-cache and allocation-census rows, a layout probe whose pins follow the cache-capacity members, and the source-bounds inventory for the new allocation sites. The two ledger cache sweep tests are pinned in the verifier's slow-label bound. docs/pages/memory-management.md, abi-stability.md and report-schema.md describe the policy, the bypass and the diagnostics; the doc anchors that moved are re-pointed.

Retire the superseded documentation sentences introduced by 819bbac, 197646c, 536f8a7, bd0cc60, 7f14930, df7ddc4, 29a825b, a94c55f, 1654497, 012f70b, cf20c17, faf4977, 6d2bf95, a991b4e, 8713a82, 6963801, 5691220, d093560, 1dd5430, f0b6213, 772766a and b4a2c19 in docs/design/ta-window-cache.md, docs/pages/memory-management.md, docs/design/tail-page-windows.md, docs/feed-projections.md, docs/design/arena-bound.md, docs/design/o1-report-contract.md, docs/design/native-feature-parity.md and docs/adr/0001-kernel-adapter-boundary.md. This explicitly acknowledges their retired no-C-setter and fixed-residency claims, intermediate cache/observer implementations, earlier payload-only metadata accounting and three/five-decorator K formulas, old ledger bounds and constructor-policy wording, the journal's former one-page EFBIG refusal and keep-window description, the retired statement that the verifier test pins the same 118/98, the ledger-cache sentence that writes fill only an existing cache, and the feature-parity design's reference to the ADR section on current state versus target, its note that the independent-host examples option is inert, and its wording of no top-level examples target, and the command-scratch fixed-K formula with its 202,752 B figure and the kCacheBytes page-size sentence (replaced by the shared read-through cache accounting); it also acknowledges replacing "Prefix configuration does not alter rebuilding or query ordering." with "Cache configuration does not alter rebuilding or query ordering." The replacement text documents the shipped per-handle cache setter, shared read-through adapters, per-kind accounting and multi-chunk journal records while retaining backing-read, allocation-bypass and deterministic-result guarantees. In docs/adr/0001-kernel-adapter-boundary.md also replace "nearest-tick rounding, ties away from zero" with the implementation's floor(price / tick + 0.5) rule without an epsilon, and name cache_policy.hpp alongside the other kernel-only includes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the configure-time check that a test carrying the fast label must be listed in PINEFORGE_PR_FAST_TESTS. The merge applied without conflicts and changes no file of the cache work.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the jailed temporary-file witness test and the strategy runner changes that landed on main. The merge applied without conflicts and changes no file of the cache work.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… no longer carry the multi-page event

A single serialized journal event larger than one spill page is refused with EFBIG and stays resident again, exactly as on main. This removes the extent writer and reader added with the cache work (ExtFile write_pending and publish_append, the ByteWriter extent constructor and finish, the ByteReader extent constructor, the 64-bit remaining count and its representability checks), the continued-page selection and validation in the journal store, the test-only resident-threshold seam and set_journal_residency_for_test, the test that drove oversized events, and its registration. A separate change implements oversized events on its own.

Everything else of the memory caches stays: the tail-window read-through caches and their statistics, the per-kind capacity policy, the successor-root cache, and the ordinary page-index arithmetic hardening.

docs/pages/memory-management.md again states the one-page refusal in the places that described continuation chunks. Retire the sentences introduced by 878af53: "Older events read back through four decoded slots using 32,768 B transfer chunks" (kSpillPageBytes), "A larger serialized event continues across as many chunks as it needs, with one directory entry for the full event", "Page size does not limit event size; scratch backing and address-space exhaustion retain their typed failures", and "An event larger than the 32,768 B transfer buffer is stored across continuation chunks and does not fail merely because it leaves residency". They are replaced by the one-page EFBIG refusal of main.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…scratch directory, and none for a small default run

Add a Linux source-layer test that watches two directories for the creation of projection backing files (names starting with pineforge-feed-): the scratch directory configured through the report sink, and the default scratch location (TMPDIR). A run with a configured scratch directory and projections one row past their fixed transfer page (the recorded series and the three calendar projections of a daily feed) must create all four backings inside the configured directory and none in the default location. A default run with one-row projections must create no projection backing at all, because a projection that fits one fixed transfer page needs no file.

Every row read back must equal the row supplied: the recorded series bit for bit, and the calendar rows against the values derived from the input stamps. The row counts come from each projection's own page size.

Nothing in src/ or include/ changes; the test is expected to fail on the current tree because the projection builders ignore the configured directory.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…a one-page projection out of scratch

The calendar day partition (trade day, week open and month open columns) and the raw-array recorded series built their projections with the default scratch directory, so a configured directory was ignored and every nonempty projection opened a backing file even when it was far below one transfer page. The engine now hands the report-sink scratch directory to those builders.

FeedProjectionBuilder gains an explicit resident mode used only by those production sites. A projection that never passes its one fixed transfer page (4096 / sizeof(Row) rows) is published from an in-memory reader of exactly that page and opens no file. The next append past the page spills through an unlinked ScratchFile in the configured directory; a directory that cannot hold the file raises the existing auxiliary scratch I/O error, with no fallback to another directory. The default builder is unchanged and stays file-backed, so the tests that prove cold reads and cache behaviour through the scratch provider keep their meaning, and a reader supplied by a caller keeps the directory it was built with.

BacktestEngine gains a protected const accessor for the directory, defined inline so no source line that the documentation anchors cite moves, and build_native_day_partition takes the directory as a trailing defaulted argument.

The new test watches the configured and the default scratch locations for projection backing files: four backings in the configured directory and none in the default one for projections one row past their page, none for one-row projections of a default run, and a typed ENOENT with nothing created elsewhere for an unusable configured directory. Row values are read back bit for bit.

docs/feed-projections.md and docs/pages/memory-management.md describe the resident page and the spill directory. Retire the sentences introduced by a279666 in docs/feed-projections.md: "The default producer writes complete rows sequentially to an unlinked ScratchFile, with one mandatory transfer page per producer", "Publication explicitly finishes the final partial page", and "Destruction performs no flush"; the production builders' one-page resident rule replaces them.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the engine changes that landed on main since the previous merge. The merge applied without conflicts and changes no file of the cache work.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…t and the feed projection change

The reviewed site inventory and the bounds manifest return to main's entries for the ByteWriter owning buffer, the journal payload append and the ByteReader cursor, which the restored one-page journal path no longer changes: the ByteReader buffer and cursor assignments and their manifest entry are gone again, and the journal payload append is back in its function scope. The six site identities of build_native_day_partition (tz, stamps, the three projection builders and the late-open assignment) carry the new trailing scratch-directory parameter in their scope.

Five documentation anchors into src/native_order.cpp (the successor-root row assertion and tail bound, index_committed, remaining_path_delivery, check_execution and apply_execution) point at the lines the cut leaves them on.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The builder now creates its scratch provider either in its constructor (the default, file-backed mode) or in a spill helper once a resident projection passes its one transfer page, publishes a resident provider holding a copy of at most that page, and keeps the scratch directory it was given. The reviewed site inventory records the two provider allocation sites, the resident provider allocation and the directory string, and the bounds manifest gains an entry for each of the last two; the existing provider entry points at its new line. Every entry stays open: nothing here claims a total-row bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The previous records commit added two open manifest entries (the resident one-page provider and the builder's scratch directory string) while the restored journal path removed one; the open unreviewed ceiling rises from 1340 to 1341, which is exactly the number of entries now in the manifest. Every entry stays open and no bound is claimed. The ceiling stays below the one on main.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ameters, default unlimited

The runner refused more than 256 symbols or feeds and the two inventory
readers refused an inventory over 8 MiB, each as a fixed constant. They are
now options: --max-symbol-feeds on the runner, --max-inventory-bytes on the
runner and the inventory binder, forwarded by the entrypoint through
PINEFORGE_MAX_SYMBOL_FEEDS and PINEFORGE_MAX_INVENTORY_BYTES. Unset or 0 is
unlimited, a positive value is the limit, and anything else is a usage error.
With a limit set the typed refusals (too_many_symbols, too_many_feeds, binder
exit 10, the incomplete-inventory admission refusal) are unchanged and name the
actual limit.

The former values live in one opt-in set, selected by --tv-compat or
PINEFORGE_TV_COMPAT; an explicit limit, including 0, wins over it. Without it
nothing is capped. The key-length limit is a name check and is unchanged. The
two new runner options share the "--m" prefix, so the ambiguous-prefix
diagnostic now lists them too; its test pins the new text.

Documentation pages and anchors that cited the moved lines are updated. The
BARS-BIN-INPUT design note drops statements that no longer hold, named here so
the revert check can match them: "All source coordinates below refer to that
tree." ; "At most 256 symbols and 256 total extra feeds across the index." ;
"`_SYMBOL_FEEDS_MAX` docker/run_json.py:N;" ; "The old comment example lists
only CSV feeds; it is an example, not an extension restriction (comment's
`feeds` docker/run_json.py:N)." ; "The actual branch accepts binary feeds
(`suffix` docker/run_json.py:N)." The sentences were introduced by 91353de; the cite `_SYMBOL_FEEDS_MAX` is too
short to match by words, so it is named by that commit here as well. The first is replaced by a statement that
runner and entrypoint coordinates follow the working source; the second and
third by the parameter wording (default no limit); the last two are folded
into one sentence saying the loader accepts CSV and binary feed paths.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The real-tree gate test pins the number of open manifest entries as unreviewed plus pending-flat-memory. The manifest now holds 1341 unreviewed entries (the ceiling raised in the previous commit) and 184 pending-flat-memory entries, so the pin is 1341 + 184 = 1525, which is what the gate counted on the real tree. The pin is derived from the manifest ceiling and is not a typed observation: every other assertion of the test is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the run-quota admission of scratch-file growth and of the sink assembly's temporary storage. Three files conflicted and are resolved by hand: the source-bounds manifest keeps the cache work's entries and takes main's three new entries under fresh ids (the cache work already uses the ids main took for its own new entries), with the open unreviewed ceiling and the gate test's pinned open count both set to the manifest's 1344 unreviewed plus 184 pending entries; docs/pages/memory-management.md keeps the cache wording and takes main's line numbers for ScratchFile::open and kCacheBytes in the rewritten scratch_file.hpp.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Takes main's run quota admission change; no conflict.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the engine changes that landed on main since the branch was cut. The merge applied without conflicts and changes no file of the exposure fold.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…efusal row

The memory-cache change had rewritten the journal limits row of this test to expect that a single event larger than one spill page is accepted across several pages, and added a --journal-limits-only entry for it. That behaviour is not part of this branch any more (the journal keeps main's one-page EFBIG refusal), so the row failed in every full verification profile (test_parent_terminal_scratch, release, debug, kernel and sanitizers) with "native o1 journal event exceeds the spill page byte bound: File too large". The file is restored to main's text, whose row asserts exactly the refusal, that nothing is published by the failed append and that a retry also publishes nothing.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Registering --max-symbol-feeds, --max-inventory-bytes and --tv-compat adds
them to the candidate lists of the runner parser's ambiguous-prefix refusal:
--t now also names --tv-compat, and --m and --ma also name the two limit
options. The exit status, the failure code and its args are unchanged. The
changelog now states this, and a test pins the exact new text of the --t and
--ma refusals (the --m text is pinned by the existing prefix test).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
When a cohort has more than 128 mapped positions the fold abandons the fixed stack sort and completes through a walk over every retained roster member. A pinned retained prefix can keep that member count growing with bars while the number of rows and mapped positions stays fixed, so the completion cost grew with history.

The overflow path now counts every mapped position without extending the fixed 128-entry workspace and compares the retained member count M with the selector's own charge (P + 1)(R + P), where P is the mapped positions and R the live rows. Below that charge the roster walk is cheaper and is used; at or above it a private copy of the original selector runs, term for term, so the acceptance-ordered floating-point additions and the returned double stay bit-identical. The reported steps include the gather and the chosen completion. The charge comparison saturates at the diagnostic integer maximum so it cannot wrap; it does not cap any input and allocates nothing.

Two rows are added. The work test builds two windows with P = 129 and R = 1 whose M differs by a full selector charge and asserts the step difference stays within the derived P(P + 1) bound (the old path differs by the whole charge, 16,900 steps). The fold test compares a P = 129 window with M above the charge against the old fold bit for bit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Takes the output series limit parameter that landed on main. The runner
parser now parses through the output-series helper and then applies the
compatibility defaults of the limit options; the changelog keeps both
entries; the memory page keeps main's text with the run_json.py citations
re-pinned to the merged file.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…der and boundary

The overflow completion's private copy of the original selector declares a next and a positions variable and a next inside its position loop; the reviewed site inventory did not know them, so the source-bounds gate named three new assign identities and the tests that run the gate on the real tree failed with it. The three records carry the scope text and ordinal the gate printed. No manifest entry changes state and none is added, so the entry and unreviewed counts stay as they are.

The rows that guard the overflow path were too weak to catch an order or filter error in that copy: one live row of 1.0 units adds exactly in any order, and a path value of 3 is what both completions return. A new fold row uses four live rows with a 1e16 row among ones, derives the only double an acceptance-ordered sum can give (a different order gives another value), cycles NaN, both infinities, negative zero, denormals and the largest double through the finite and positive filter, and pins which completion ran from the step count at M one below, at, and above the selector charge. The work row now also asserts that two windows differing only in M cost exactly the same steps and that the cost stays within (R + P)(P + 2).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The output series limit change raised PF_STREAM_ENGINE_LAYOUT_VERSION from 1
to 2 on purpose (the engine gained a member), and the module-layout assertion
in native_live_startup_e2e.py still pinned version 1, so the native-live legs
failed on main. The pin moves to 2: the normally linked native example must
still export the stamp, and it must still be exactly the version the runner
accepts; an accidental bump stays a test failure.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… with their rows

The fused journal fold replayed the old validation scan only when the backing read failed after a skipped text. A schema failure found by the fused pass in the same record was returned as such, although the old two-pass code would have met a read fault on its next page first (its refill boundaries differ after a skipped text), so the status and diagnostic of a record that is both corrupt and unreadable changed. The fold now replays the old scan on any fused failure after a skipped text, still never on a successful record, and keeps the original failure when the replay finds nothing.

In the bounded-batch exit scratch builder, a run-identity failure raised while an earlier append had a batch open unwound through the scope's teardown with the transaction still active, so the teardown began a nested transaction, remembered a programming error and left the transaction open. The builder now rolls back an open batch before raising that failure and on destruction (before the scope erases its rows), finish() releases the batch and rolls it back if its teardown check fails, and a failed put or commit keeps the builder's batch state in step with the store. The class comments no longer claim that every edit commits separately.

New rows: the journal fold across fault offsets after a skipped text compares its status and diagnostic with the old composite built from the old functions; the exit scratch rows cover a run-identity failure at a count that is not a multiple of 64, a put or commit fault in the middle of a batch, and the 63-row boundary, each comparing the exception and the end state with the per-row path.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 and others added 11 commits November 1, 2026 00:38
… throw allowlist

The batch builder's finish() and fail_batch each roll back or record the state of an open batch and rethrow the exception they caught; the run-failure code check reads a bare rethrow as an uncoded throw unless a reviewed row admits it, so it failed, and the verifier tests that run the real check failed with it. Add the row for the two statements, with the reason: the original exception keeps its own code. No throw is added or changed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…the curve read cost and the jailed projection demand

A spilled equity curve went through the shared read-through cache with one 24-byte point per line and no read-ahead, so a forward or reverse pass over N points cost N backing reads where the previous 64 KiB window cost ceil(N / 2730) (a lab run: 5,477 points took 5,478 backing reads against a bound of 5). When a backing miss is adjacent to the previous lookup (the next or the previous point) and the cache is enabled, the window now reads the aligned block of up to 2,730 points containing it with one read into one fixed array of 2,730 points in its state, and serves the points of that block from it. Any other miss reads one point as before; a lone point is kept until a sequential block replaces it, so a distant probe cannot lose it. A failed block read publishes nothing. Zero and sub-point capacities and the allocation-failure bypass read each point directly and never touch the array. The array is part of the window's state, so cache_control_bytes() already charges its 65,520 bytes; nothing scales with points. The default 2,730-line policy, the policy setter, every value and every metric bit are unchanged.

Tests: the curve test counts the backing reads of a forward pass, a reverse pass (also at 2,731 and 5,461 points, where the endpoint probes collide in the default cache) and asserts ceil(N / 2730) plus the two boundary probes, keeps the zero and sub-point capacities at one read per access, bounds the random access byte total by 24 bytes per lookup plus one block for each adjacent pair of the generated sequence, compares every returned point bit for bit, and checks that the extremes fold stays exact. The ordinary heap budget of the selected-metrics test adds the block workspace. The feed projection test gains a jailed case: with an arena attached, a calendar projection and a recorded series longer than one page succeed inside an arena sized from the page and chunk arithmetic, create no file in the configured or default directory, and fail with the arena's typed exhaustion error when the arena is one byte smaller.

Docs: memory-management.md, tail-page-windows.md and feed-projections.md state the block fill, the fixed workspace and what cache_control_bytes() includes, the in-memory limit of one 4096-byte page per calendar column or recorded tape (512 int64 rows or 256 recorded rows of 16 bytes), and that an attached arena supplies the spilled pages; two stale line citations are re-read from the source. Sentences replaced by that wording, named for the revert check: "Each point is a 24-byte shared-cache page." and "The curve therefore adds one metadata charge per point, rather than pretending that all 2,730 records share one tag." (a279666); "Metrics re-read scratch through bounded windows" and "`ScratchCurveWindow` src/report_scratch.cpp:N)." (ff05c78); "2,730 complete 24-byte points; the writer's retry batch is mandatory workspace" (645e8a3); "Production builders keep projections of at most one fixed transfer page in an in-memory reader and spill larger projections" (9edaaa7).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The memory-cache head adds the shared read-through cache, the per-kind cache policy, the cache statistics and the policy setter to the C API; the runner-limits head adds the output-series limit functions and moves the engine layout stamp to 2. Both sets of exports and both sets of documentation survive: the runtime inventory is 122 declarations with 102 implementations (the common base's 118 and 98 plus two from each side), pineforge.h declares 111 of them, and the checker, the exporter test, the driver test and the layout probe expect those counts and the combined sizes (the adapter 89,488 bytes, the host 107,944 bytes, derived by hand from the merged headers). The pages that cite docker/run_json.py, docker/entrypoint.sh or the report scratch source take their line numbers from the merged files.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ster size

The choice between the roster walk and the selector copy compares the retained member count with the selector charge. Every row of the overflow path built a roster whose retired prefix is empty, where that count equals the total number of accepted entries, so using the total in its place went unnoticed. In a run, retiring a prefix makes the two differ: the results stay bit-identical but the work does not, and the wrong count picks the selector when the walk is cheaper.

The new row puts two unmapped entries in front of the mapped block, pads the roster to one more than the selector charge, retires the two entries and so leaves the retained count one below the charge while the total stays above it. It asserts the retained and total counts, the sole large row's absolute position, the order-sensitive total, and that the walk ran (the gather's steps plus the retained count); a threshold on the total would report the selector's steps instead.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The commit makes a sequential pass over a spilled equity curve read whole 2,730-point blocks again, with the tests and documentation that pin it. One documentation paragraph was edited on both sides: the train's copy carried the shifted line numbers of the writer constants, the commit's copy carried the new description of the reader. The merged paragraph keeps both: the reader description of the commit and the line numbers read from the merged sources (the constants at report_scratch.hpp:20-22, the writer state at report_scratch.cpp:160-168, the reader state at :278-334).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Two conflicts. The run-failure throw allowlist gains both sides' rows (the series window's rethrows and the exit scratch batch builder's two), each row unchanged. The native-feature-parity page was edited on both sides only in its line citations; the train's text is kept, and the citations into the adapter source that sit after the two lines this change inserts (one near line 25,856 and one near 26,372) are re-read from the merged source by the documentation anchor check that follows.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…and two documentation fixes

The window clears its block before it reads a replacement and publishes the new one only after the read completes, so a read that fails after a short read cannot leave a half-overwritten block marked valid. No row exercised that: the existing failed-refill rows never reach an adjacent block fill that fails. The new row walks the first block through a one-line cache, shortens the backing file inside the next block, expects the typed I/O failure on the adjacent lookup, and then requires an intact early point to read back bit for bit (without the reset it would return the bytes of the partially overwritten block).

The equity writer's citation in the arena bound page pointed at the line that is now another function; it names the writer's write line. The cache parameter page now says what the documented bypass costs: zero capacity turns the cache off, so each lookup reads one point.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The exit scratch batch change inserts two lines into the adapter source ahead of the cited symbols, and the train's other members moved the reflection helper in the admission source by four lines. The documentation anchor check named five citations whose symbol had moved: the pending-order birth reach (two rows, 27807 to 27809), the fill-quantity probe (27386 to 27388), the risk direction setter (27193 to 27195) and the reflection field helper (658-661 to 662-665). Each now names the line that holds its symbol in the merged source.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Main now holds the runner limits change as one squash commit whose tree is identical to the head this train was built on (the tree of the limits branch's last commit and the tree of the squash are the same object). The train already contains that head and its history, so the merge keeps the train's tree unchanged; a content merge would only re-apply the same lines against the train's later edits to the same documentation hunks and conflict without changing the result.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Code graph out of date: Health delta baseline: last indexed commit 78f369e, 1 commit(s) behind this PR's base. Impact and hotspot numbers below may be wrong.

No verdict: the code graph is out of date. Static checks found no blocking issues, but they compared against an outdated graph. The next push after the repository re-indexes gets a full review.

Not checked: tests were not run; no formal proof of the changed code.


Graphify review — findings

Adds strategy_set_cache_policy_v1, which lets a host set per-kind resident cache byte budgets while the handle is idle. Kinds left out keep internal defaults, a zero budget bypasses that cache, and changes take effect from the next run. Adds pf_report_cache_stats, which reports per-handle resident command-cache and ledger bytes plus a count of optional allocations that fell back to bypassing the cache. run_json emits cache_bypass_reason: "alloc_failed" and allocation_bypass_count only when that count is positive. The grid-search benchmark now binds the policy entry point, and the README export counts and the source-bounds ceiling (1346 → 1344) are updated to match.

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 17856 functions depend on the 13032 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: session_period_close_ms() — 26 callers, 6 callees
  • worse: session_period_open_ms() — 30 callers, 5 callees
  • worse: begin_ready — 4 callers, 21 callees
  • worse: session_period_last_traded_close_ms() — 8 callers, 8 callees
  • worse: field() — 8 callers, 7 callees
  • worse: session_day_index() — 18 callers, 3 callees
  • worse: _selected_run() — 1 callers, 47 callees
  • worse: build_native_day_partition() — 9 callers, 5 callees
  • …and 79 more — each is listed as a finding

Verification — 17856 functions in the blast radius were not formally verified this run (proofs are advisory here).

Health delta baseline: last indexed commit 78f369e, 1 commit(s) behind this PR's base.

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 17538 function(s) in the blast radius were not formally verified this run

Docs that may be stale (advisory)

@luisleo526
luisleo526 merged commit 69cddc9 into main Oct 31, 2026
15 checks passed
luisleo526 added a commit that referenced this pull request Nov 1, 2026
…30961 dropped from the bars-bin input page

Two spots of docs/design/BARS-BIN-INPUT.md in this branch no longer
carried the text main has, a side effect of resolving the merges of
main commits 69cddc9 (#679) and aa30961 (#675). This commit puts main's
text back; nothing else changes.

- Feed input/output refusal row, introduced by 69cddc9 (#679): the
  converter reason list is main's list again ("`feed_columns_missing`,
  `feed_value_not_number`, `feed_value_invalid`, `feed_time_out_of_range`,
  `feed_close_time_invalid`, `feed_unreadable`, `feed_not_utf8_csv`,
  `empty`, `count`, `io`, `timeframe_invalid`"); the branch had replaced
  it with an older, longer list that carried an extra reason. The row's
  code pins stay as merged (the `return` pin is docker/run_json.py:5267).
- Selected-window comment sentence, introduced by aa30961 (#675): "The
  `1.7.0` wording in the loader's selected-window comment is not an
  emitted release version (`_request_invalid` docker/run_json.py:4948)."
  The branch had replaced it with the older "comment's `QC6a`" wording;
  the pin is main's pin renumbered by one line for the merged source.

Verified: git diff against main for this page now shows only the pin
renumbers that the merged source needs.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Nov 8, 2026
…n, trade text and RCI changes (#788)

## Summary

This branch combines seven source heads into one change on top of main (`755e6658`), so that the build, the parity sweep and the review each run once. Every source head was merged with an ordinary merge commit: nothing was rebased, nothing was squashed inside the branch, no conflict occurred in these merges and nothing was resolved by hand. Every source head contains main (`755e6658`) except the trade text channel's, which contains the main before one documentation-only commit (`63f3978a`); the combined branch contains main. When this change merges, pull requests 781, 758, 780 and 783 are closed with a link to it.

A moving average change (head `4c5c65f4`, "Compute SMA from a correctly rounded window sum") was merged into this branch between the trade text channel and #783 and is taken out again by the last commit of the branch, a plain revert of that merge (`git revert -m 1`, no conflict). The parity measurement of the branch with it showed 40 probes whose graded tier falls against their committed TradingView tapes, every one of them calling `ta.sma`; a measurement of main plus only that change gave the same 40 results, and no other probe changed. The change is not part of this merge; whether it returns, and in what form, is decided separately. The tree of this branch equals the tree of the seven heads below merged on main without it.

PR type: code (engine sources, tests, CI scripts and documentation).

Fixes #779

## Sources

| Source | Title | Source head |
|---|---|---|
| #781 | ci: validate split releases and retained smoke output | `130bda2db570bd63bdec57b6b5fdd3cabacab7fc` |
| branch | Validate corpus records and gitlink results in the pull-request checks | `46bb21fb169091722d49d3b2f2e4f1f589148166` |
| #758 | test(ci): register rerun-preempted controller tests | `348a89766e0ff23f88b4837bce80957f7635ef8e` |
| branch | Add an opt-in trade text companion channel | `5930d7f8d12010d67c77a098804eb6cd295095db` |
| #783 | Add generated CLI coverage for map pair limits | `163abf58f1a73c7ec9160120774954fb29f4bd2d` |
| branch | Avoid copying RCI price history during ranking | `022750f8451aadd93066ffad3b54695e23956116` |
| #780 | Make nested consumer state construction portable to Clang | `000e2dac0e8e5e73fe53551adec7fbdf3f251af6` |

The head listed for the corpus validation change already contains the heads listed for #781 and #758, so merging it brings both along and the separate merge of the #758 head changes nothing (it is listed because its pull request closes with this change). The merges that this branch makes of the source heads resolve no conflict by hand. The source histories themselves carry six merges that were resolved by hand in their own branches, before the heads were handed over: three in the history of the corpus validation head and three in the history of the trade text channel head. Corpus validation: `d482eb37` (`scripts/test_ci_preflight.py`: the stage-inventory counts between the registration of the rerun-controller tests and the corpus validation job), `f7519bb8` (`scripts/ci_preflight.py` and `scripts/test_ci_preflight.py`) and `cecedc0e` (`CHANGELOG.md`, `ci/cr150-source-bounds.yaml`, `scripts/test_cr150_source_gate.py` and `tests/private_fixture_reads.evidence`). Trade text channel: `09c9fb52` (`CHANGELOG.md` and `runner/main.cpp`), `8c17d14e` and `5930d7f8` (both in `ci/cr150-source-bounds.yaml` and `scripts/test_cr150_source_gate.py`). The head listed for #758 is that pull request's branch merged once with main; the head published on #758 is older. The head listed for #780 is the head of its branch (the construction fixes and the refreshed source citations in three documentation pages).

## How it was built

The merges were made in the order of the table, with the moving average change between the trade text channel and #783 (see above), except that the separate merge of the #758 head was skipped because the corpus validation head contains it. Relative to main, 52 files change. Besides the files that #781 and #758 share with the corpus validation head, three files are touched by more than one source and were merged automatically without a conflict: `docs/ci.md` (the corpus validation change and #783), `docs/design/native-feature-parity.md` (the trade text channel and #780) and `scripts/test_ci_preflight.py` (#781, #758, the corpus validation change and #783). For every source, the changed lines of the files that only it touches equal the changed lines of this branch, and for every shared file the union of the sources' changed lines equals the branch's; an independent chain of the same merges in the reverse order gives the same tree.

docs/ci.md: the slow-label bound paragraph and the slow-label inventory
paragraph, introduced by 69cddc9 (#679), now state the size of the pinned set
(90 names, including test_cr559_map_capacity_cli) instead of the retired
43 + 28 = 71 breakdown.

docs/ci.md: the three sentences of 42527b6 (#704) that said gke-pr.yml admits exactly two variants, that OLD is the shape without the arm job kept so earlier workflow files still pass, that NEW requires the exact arm classifier output, and that hybrids and extra jobs, outputs, needs or arm steps are refused while OLD retains its original exact rules and templates for a workflow without the arm job, are rewritten to cover the arm and the corpus variants.

## What was done before merge

Hosted CI was green on this head. One parity sweep against the active baseline measured this exact tree: 7,989 probes compared, no probe changed tier, no leaver, no regression, and the gate returned its neutral pass. One review of the combined diff, which cites each source, and of the revert commit gave a GO on this head and tree. The results already recorded on the individual pull requests and branches are not repeated here and do not stand in for these. The moving average change is not in this merge: it was merged into this branch and taken out again by the last commit, a plain revert.

The full post-merge workflow was also run on this head (run 42452226570). It failed exactly the twelve jobs that fail on main `755e6658` (run 42435140307): the eight sanitizer shards and the live-sanitizer job, each stopping at the final link with "No space left on device", and their aggregate jobs. Every other job passed, and no job failed that passes on main.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant