Combined 1.8.0 batch: scratch caches, live-runner reports, adapter scaling, UDT graphs, metrics replay and the CI admission fix - #616

…native capture tool and a drift test

Foreign readers of the report-sink stream (a worker-side TypeScript reader, job
rows) cannot run docker/pf_sink.py. docker/gen_sink_vectors.py writes one
versioned set of byte-exact streams. The Python vectors are produced by the
Python test fixtures that mirror the writer's framing, not captured from the C++
writer: a complete stream with a record of every kind (E, T, O, V, K, X, H) and
the trailer, two streams with a bad trailer (counts, sha256) and two truncated
streams (inside a record, at a line boundary), each with the expected result
recorded from the engine's own reader (the parse as JSON, the converter's
output, or the exact exception class and message). Their header carries
placeholders (empty run_id, engine "test"); the page lists the header fields a
real stream fills. docker/capture_native_sink_vector.py runs docker/run_json.py
on a built hand-written strategy library and a small bar file and saves the
stream exactly as the C++ writer wrote it, with the reader's parse and a
provenance record; the generator's --native-dir adds those files to the
manifest, which carries the wire version, the engine commit, the generator
command and every file's size and sha256.

tests/data/sink_vectors_v1/ holds the set. docker/run_json_sink_vectors_test.py
regenerates the Python vectors and compares them byte for byte with the
committed files, re-reads every committed stream with the reader and the
converter, and for the native files checks the manifest hashes, the reader's
parse and a real header. docs/pages/sink-vectors.md describes the files, what
the set does not cover, and how a foreign reader uses them.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Bring the branch up to date with main so its checks run on the current tree.
No conflicts; no file of the branch changed by the merge itself.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ith a doc-lint rule that holds it

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… tree builder reads

CiFileLoading.make_tree copies the actionlint config and tests/CMakeLists.txt
using self.CONFIG and self.CMAKE. test_second_workflow_files_are_discovered_by_the_ci_entrypoint
lends it an AppTokenEnvironments test that had neither attribute, so the
preflight-tests stage failed with AttributeError before any assertion ran.
The test now carries the same two class constants; no assertion changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Bring the branch up to date with main so its checks run on the current tree.
No conflicts; no file of the branch changed by the merge itself.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…in the number of bars

Strings and maps now live in the paged object store instead of the C++
heap, so their memory does not grow with the number of bars a script runs.

- ScriptString and the map image keep their bytes in store objects; split,
  concat and the map operations read and write through the store.
- Store transactions use a fixed pool of handles (four caller slots plus one
  cleanup slot). Running out throws TxPoolExhausted before any state
  changes; there is no heap fallback. Store::scope() joins an active
  transaction without taking a slot.
- The join hooks join_active_tx() and abandon_joined_tx() are public pure
  virtuals so forwarding wrappers can pass them on. begin_cleanup() is used
  only by the map destructors, and note_cleanup_skipped(CleanupSkip) records
  whether a cleanup was skipped because the transaction was rollback-only or
  because it failed.
- A store refuses the (limit+1)-th live object with
  scratch_capacity_exceeded. The limit is Shape::max_live_objects, default
  and ceiling 1,048,576, counted per store and rolled back with the
  transaction. The error reuses the cohort resource name for now; a
  dedicated name follows before the 1.8.0 tag.
- Tests cover string and map parity, rollback, nested destroy, split, the
  pool limits, the live-object cap and wrapper forwarding, plus 100k and 1m
  growth runs for strings, maps and prepared maps.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Plain merge; keeps the strict ruleset's up-to-date requirement.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ng records

CommandScratchStore::advance_handles(count) reaches the same logical
allocation state as count successful new-key puts followed by erasing
those records: live handles, bytes, count and ordered digest are
unchanged, the state digest includes the advanced allocation counter, and
the work is independent of count. Handle exhaustion fails at the same point
as puts. A property test covers equivalence, transactions, rollback and
the overflow edge.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… every historical id

A reversing entry cancelled the exits of every cohort id the run had ever
seen, which made each reversal cost grow with the number of distinct entry
ids. The cancel now visits only ids that can own a queued exit (pending
bracket legs, COOF requests, shadow rows, relative and anchored legs, live
exit rows), keeps the old sorted owner order, reserves the same scratch
handles through advance_handles, and keeps erasing empty bracket families.
Debug builds check every skipped id against the old per-store predicates.

A public-run test compares the gated path with the retained old loop on
five owner cases (live, empty, COOF, anchored, pending-bound): trades,
report and equity bytes, hash reads and reversal count match, and the
cancel-call count stays zero as distinct ids grow.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

The gate tested cohort membership with find(), which runs the implicit spill pass and restores a spilled row into the resident set. DiskResidentIdTable::contains() reads the index alone, so the gate no longer restores or spills any cohort row.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Two conflicts, both in the source gate. The scanner change here retires the entry for the
trade-ledger iterator index (it no longer matches a site), so that entry stays removed while main's
new entry for the command scratch cache budget is kept. Main also added two entries that reuse the
ids this branch had given to the object-limit message and the map iterator temporary; those two
move to U1545 and U1546. The unreviewed ceiling becomes 1281 (1279 here plus main's 2), the
pending-flat-memory ceiling stays 185, and the pinned OPEN count becomes 1466, the number of
entries (1463 on the common base, plus 2 from main, plus 23 added and minus 22 removed here).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… drop the stale 180 s wording

test_selected_window_ownership_manifest_controls_shard_1of4 took 74 s on an idle
8-vCPU box while shards 2-4 took 41-50 s (the modulo split over sorted case ids
put the heaviest controls together). The four rows carry no explicit TIMEOUT
any more (#504 removed it) but the tree still described a 180 s budget and the
CMake guard still exempted the exact old statement.

- Shards are now assigned largest-first to the lightest shard, with a weight per
  case read from its AST (checker subprocess launches, through self helpers and
  literal loops). No measured seconds are stored. The partition stays exact,
  order independent and deterministic; --shard I/N validation and the
  "ran N tests as selected" fail-closed check are unchanged.
- The two 60 s subprocess deadlines inside the controls are removed.
- scripts/check_cr150_cmake_d.py no longer exempts the old six-row
  "TIMEOUT 180" statement, and its test now requires the refusal.
- docs/ci.md, tests/ownership/README.md and the CMake comment describe the
  balancing and no deadline.
- New rows: balance bound (max/min <= 5/4, violated by the old split) and an
  estimator row; the partition row now checks input-order independence.

Documentation sentences removed from docs/ci.md, all introduced by 375795a (the
change that split the ownership manifest controls into four CTest shards): the
sentence saying that change splits the controls into four CTest shards
(`test_selected_window_ownership_manifest_controls_shard_1of4` through the
fourth), the sentence saying it adds three public registered/run rows to every
engine profile in both fixture worlds and only raises the counts above the
pinned minimum floors, and the sentence saying the private-fixture and receipt
row populations stay the same. The replacement paragraph states the balanced
four-shard split and the unchanged row populations.

Measured on a throwaway 8-vCPU spot VM: shards 51/57/53/55 s after, 74/42/41/50 s
before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

test_private_fixture_reads pins a token at tests/CMakeLists.txt:3301. The two new registrations added two lines above it and failed that check. The reversal-gate row now replaces one comment line it folds, and the bulk-advance row joins the existing report-failure registration, so the file keeps its line count.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…er-platform stream memory bound

The shared descriptor guard compares every status and descriptor flag at each
boundary and masks only Darwin's kernel-maintained "descriptor was written"
bit (FWASWRITTEN) under __APPLE__. A new test plants a real F_SETFL, F_SETFD
and close after a pipe write and requires the guard to reject each; two more
ctest rows run those planted modes through a small CMake driver that accepts
only exit status 1 together with the guard's own message.

The 64 MiB stream memory test keeps its 1 MiB bound everywhere except Darwin,
where it allows 2 MiB for the 16 KiB page size.

CHANGELOG notes the macOS report-sink limit is no longer listed (re-examined,
not re-measured). 1.8.0 is Linux only, so no CI job exercises the Darwin mask
or the Darwin bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Attribute exclusive opens and fork-descendant scratch files to their creating
process, keep unrelated co-tenants out of the witness, and report the offending
path with its PID or an explicit unattributable reason. Add three public Linux
CTest cases for exclusive open, inherited evidence, and diagnostics.

Carry the three rows through verifier floors and mock accounting. Preserve
documentation line counts and refresh the floor-table citations.

Documentation replacements from 3ea5a0d name the old wording:
"Their default runs now enter strict exclusion accounting"
"Verifier 8 + preflight 8 + command-after 8 + report 4 add 28 public rows; plain Debug-type runs enforce the increased registration floors."
"Release 938 with the overlay or 631 without it"
"The floor itself remains fixed at 631/334 absent or 938/365 present"
"The native PR registration floor is 1000"
"Both numbers are constants of the profile"
"The full-run release and kernel floors are the table's own numbers above"
"EXCLUDED_REGISTERED_MIN holds 904 for debug, 895 for sanitizers"
"The run floors in the table above"
The release and kernel table rows from 3ea5a0d use the new floors.

The population wording and table header from b97c409 are replaced:
"The registered column is why the absent floor is not the absent registration, and it is to be re-measured after the registration fix lands."
"registered (present / absent)"

The four citation edits retain the passages named by:
"The floors' numbers live in one place"
"Each profile counts the rows that"
"the minimum number of CTest rows"
"The profile carries a row floor"

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Plain merge; keeps the strict ruleset's up-to-date requirement.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…inal docs wrap

The App-token environment guard recognised only actions/create-github-app-token
by action name; a tibdex/github-app-token step fed from vars, inputs or a secret
whose name does not say "private key" passed without an environment. Both
actions are now recognised by name, with a refusal row and a removal control for
every ref and quoting form and every key source.

The docs paragraph about the deployment environments is shortened to its facts
(no internal card id) and the four neighbouring sentences keep their original
line wrapping, so the paragraph is a pure addition to docs/ci.md.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ed reviewer

Read back from the repository settings: both environments allow deployments only
from main and neither has a required reviewer, so the docs sentence no longer
claims an owner review for `release`.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Corrects four comments that no longer match the code they describe, and one
stale line number in a script comment:

- scripts/ci_verify.py: the selected-sink memory test is registered at its
  current line of tests/CMakeLists.txt.
- tests/test_scratch_create_pinned_text.cpp: the header names the files that
  produce the pinned create diagnostics instead of a release.
- src/source/pine_adapter.cpp: the placement table scans the retained
  placements and consults the historical minima for erased rows; it has no
  per-id index.
- include/pineforge/native_host.hpp: the live-order read comment no longer
  claims the reads allocate nothing, since a spilled row is read into a buffer.

No code changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… path it pins

The header named two producing files; the test also pins the id-pair scratch
file and the successor-roots create. It now lists all four scratch paths by
what they are, and drops a stale note about an unobserved prediction. Comment
text only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

….hpp

Mechanical relocation of the BookSpill implementation (about 290 lines gathered from three places in native_order.cpp) into a header of its own, included once from native_order.cpp. No behaviour change: only the order of definitions inside the translation unit moves, and nothing between the old sites uses them.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…led orders

A working order that cannot act at the current point is kept dormant instead of being evaluated at every point. Resident dormant rows are refreshed by one bounded bulk sweep in winner order, and spilled rows carry a stamp so their allowance is projected on read to the exact value the eager path would hold. Rows above 4096 spill to a disk-backed book, and the match path touches only candidates in a resident tier plus O(log n) index steps, so the cost of a point no longer grows with the number of standing orders.

Trades, fills, hashes and refusals are unchanged on every valid input: the dormant tier is an exact cache of the eager allowance (same value, same epoch, same token), any dormant-only failure degrades to the eager path, and a run that cannot use the fast path takes the eager path unchanged.

Tests: a transcript suite compares eager and dormant runs byte for byte (hashes, allowances, fills) across the seven driver classes, spill from the middle inside notification callbacks, held-copy reads, bad_alloc and read-failure degrade paths, and a cost row bounding disk reads per extra spilled row. 43 mutation patches with a manifest name the row each one must turn red. test_native_dormant_book_capacity is added to the pinned slow rows.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…labels

Reword published documentation that carried internal task and card labels so each
passage says what the thing is. Lines that only changed a label keep their meaning.

Sentences replaced or dropped, named by the commit that introduced each:
- 91353de: "Read-only census for CR-150 L22u / A6 and A5-WIDER." and
  "All eight vectors use the three-term verdict;" (arena-bound pages), and the two
  passages of cr150-contracts.md that cite the external design lane's work files
  and its revision 6 amendment.
- d093560: the paragraph of native-refactor-progress.md that quotes the supervisor
  cap and the decision that stopped the R5 performance wave.
- a77fb73: "R4-D L1 advanced BacktestEngine, NativeStrategyHost, and the private
  consumer to engine_script_run_v17;" (abi-stability.md).
- 44bb687: "R5 L4 adds NativeRunSpec::margin, folded only when it is set;" and
  "R5 L5 adds calculation, max_recalculations_per_point and open_ba..." (abi-stability.md).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…bined

Run the doc-anchors checker's own --fix on the combined tree (two citations in
docs/design/native-feature-parity.md and docs/pages/public-contract.md), and
re-point three citations it cannot fix mechanically: the BacktestEngine state
spans in engine.hpp (393-595 and 519-568, the original spans moved by the 35
lines inserted ahead of the class), execution_terms_grid_representable in
native_execution_consumer.cpp (512) and commands_allowed (1968).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Regenerate the CR150 site inventory with the tool's own output for the tree
that includes the UDT graph change, append the one seed entry the gate asked
for, set the ceilings to the counts it reports (unreviewed 1299,
pending-flat-memory 184) and the pinned OPEN count in its test to the gate's
1483. Re-point four documentation citations that moved: run_json.py
_runtime_supports_selected_window (4817) and _write_report (4773) on the
public contract page, hash_cohort_receipt (936) in the adapter boundary
record and native_intrabar_path_digest (295) in the CR150 contracts.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…r after the merge

The public contract page cited run_json.py positions from before the report
channel change: the argument table moves by nine lines and _make_state,
_selected_run and _capabilities_main by seven. Re-point them to the lines the
combined file has. Re-point the comment citation on TradeLedger::cache_resident_bytes
to include the declaration, and the engine.current_bar_ assignment in the native
engine page to its line in native_execution_consumer.cpp.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

Brings in the pull-request admission fix (release.yml's matrix runner accepted under its fetched path) so it lands with this stack.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

test_p2cx_sc412 compared the cached and uncached budget runs through images(),
which reads a registry of descriptors and inodes recorded at syscall time. A
rehash closes the old index files, and a later file can reuse both the recorded
descriptor and its inode, so the two runs could capture different files while
both tables were correct, and the bit-for-bit comparison failed (seen in the
debug and kernel profiles on the combined stack).

The tree table and the pair store now expose their files by role to the test
through private-member accessors in a test namespace, and the test hashes those
files directly, in a fixed order. The assertion text and the comparison are
unchanged. No production file is touched.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…not by every trade it has closed

The read cache was sized from the number of trades the ledger had ever closed (up to the cache row cap). A ledger that retires sealed rows refuses reads below its first retained row, so only the retained rows can be cached, yet a run that closed many trades still allocated a cache as large as the cap and kept one per-trade heap term resident. The cache is now sized by size() - first when the ledger retires, which keeps the allocation at the retained window however many trades pass through. The cache slots are addressed by the global row index masked to the capacity, so any capacity that covers the retained window holds it without collisions wherever it starts.

test_ledger_retired_cache_bound closes 196,608 trades through a retiring ledger, reads and seals each, and asserts the largest cache allocation never exceeds the retained-window bound; it fails on the previous sizing.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ing them

A failure of the heap bounds now comes with the peak, retained bytes, resident rows and cache bytes of the path and fd samples, so the term that grew is named by the run output instead of being inferred.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…che cap and to zero growth past it

A sink run keeps the ledger's read cache at its declared constant cap (TradeLedger::kSinkCacheRows rows of kLedgerEntryBytes each). The witness now bounds the sink heap by the zero-trade control plus that cap, read from the engine's own constants, plus the existing fixed slack, and requires the excess over each population's control to agree within the slack across 100,000, 200,000 and 1,000,000 trades: the cap is reached by the first and passed 15-fold by the last, so any per-trade heap term shows as growth.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

The heading carried a release version and date that no tag backs (the newest tag is v1.7.0 and VERSION says 1.7.0). It now reads Unreleased until the release is cut. The residency witness header also states that a retiring run sizes its read cache by the retained rows.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… no longer gates the 20,000-order row

test_standing_orders_wall bounded the spill reads of N pending orders to twice the population growth from its first grid point, N = 2,500. On the combined tree the reads per order are 74 at 2,500 (the run still fits the resident part there), 117 at 5,000, 163 at 7,500 and 194 at 10,000, so a reference below the resident cap turned an ordinary step from the resident regime into the spilled one into a failure, although the total reads were lower than on the previous tree at every N. The reference is now N = 5,000, the first grid point above the resident cap; the factor of two and every other bound are unchanged.

The 20,000-order row is moved out of the gating grid. Reads per order keep rising linearly with the order count (117, 163, 194, 230 and 271 at 5,000, 7,500, 10,000, 15,000 and 20,000), so the total is quadratic, and 20,000 exceeds the factor (2.30). The same term is present on the chain this tree was combined from (251, 300, 335, 378 and 423 reads per order on that chain, the same slope), so it is a pre-existing defect, not a result of this change. It is tracked in #621 and the row returns to the gating grid when it is fixed; --n=5000,7500,10000,15000,20000 still runs it. The usage text accepts up to five counts.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

The comment on cache_resident_bytes gained two lines, which moved the lines the memory-management page cites. It says the same in the original number of lines.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>