Skip to content

run_json: assemble the report from one pass over the sink spools - #590

Merged
luisleo526 merged 109 commits into
mainfrom
run-json/one-pass-report
Oct 28, 2026
Merged

luisleo526 merged 109 commits into
mainfrom
run-json/one-pass-report

Conversation

@luisleo526

@luisleo526 luisleo526 commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What

The report wrapper (docker/run_json.py, docker/pf_sink.py) now assembles the report from one pass over the native sink. Before, one run read the sink several times: a strict verify pass, a separate pass for each of the trade and equity arrays, the summary and diagnostics walks, and load_bars parsed the bar file row by row.

  • One read of the sink. It verifies the header, every record, the trailer, the native receipt and the end identity, and in the same pass folds the summary, keeps the diagnostics and writes the trade and equity arrays to bounded temporary spools. The final document is spliced from those spools in the same key order and bytes.
  • load_bars reads the bar file once in slabs (13-digit millisecond timestamps, finite decimal or exponent floats) and replays from the first row it cannot prove through the exact row-by-row reader, so odd files give the same bars, hash and errors as before.
  • Bounds. Memory does not depend on the number of trades or equity points: fixed read and hash windows, at most 128 proof rows, one bounded line, two equity endpoints and scalar folds.
  • Failures. Every refusal keeps its typed code, text, exit status and priority. A failure mid-pass destroys both provisional spools and prints only the failure document, never a report prefix. The report uses exactly the bytes that were hashed in that run, so a same-size rewrite of the sink after the read is deliberately not seen; format, trailer, receipt and end-identity refusals are unchanged.

Tests

  • docker/run_json_fused_test.py (registered in tests/cr150.d/p2cx-rjone.cmake) compares the product with a kept copy of the previous release's wrapper and sink reader (tests/fixtures/*_frozen_6b33c47f_rjone.py, byte-identical to docker/run_json.py and docker/pf_sink.py at 6b33c47f, checked by hash at import): complete report bytes on synthetic sinks and on corpus runs, every strict refusal, competing failures, atomic publication, one read of the sink, the end identity re-check, and the bulk bar loader.
  • Superseded tests. One-pass assembly no longer calls the separate trade and equity scanners that these tests corrupted through. Each is replaced by a test that puts the same corruption into the finished sink file before its first read and compares the complete failure envelope, output and cleanup with the previous release's reader:
Replaced By
test_digest_refusal_discards_provisional_report_bytes test_trade_corruption_before_fused_read_discards_provisional_report_bytes
test_a_refuses_whitespace_but_c_proves_the_same_changed_bytes_by_digest (space, tab, carriage return) test_equity_whitespace_before_fused_read_keeps_frozen_refusal
test_equity_digest_refusal_discards_provisional_report_bytes test_equity_digest_before_fused_read_discards_provisional_report_bytes
test_equity_decoder_exceptions_after_a_are_typed_and_never_published (7 faults) test_equity_decoder_faults_before_fused_read_are_typed_and_never_published
test_equity_framing_exceptions_after_a_are_typed_and_never_published (2) test_equity_framing_before_fused_read_is_typed_and_never_published
test_selected_validation_reads_only_the_first_equity_record (strict, bulk) test_selected_validation_checks_the_first_equity_point_once (the old pass count no longer exists)

Not covered

test_run_json_bounded_transport::test_artifact_cache_is_bounded_on_the_deployment_backing fails on the CLOUD build hosts (the build directory is on overlayfs) with and without this change; on the unchanged tree it fails the same way (a verify run of the unchanged tree: 1 failed, 54 passed). Corpus-wide report-byte equality over 40 runs and the timing replicates (fixed-id and unique-id loops) are run separately on a spot box and posted in this thread.

🤖 Generated with Claude Code

ci-cx and others added 30 commits October 25, 2026 21:14
…ents and refuse a minting job without one (CR-304)

release.yml `release` and `notify-hub` and the release ruleset probe's `probe` job use the `release` environment (human-dispatched); promote-baseline.yml's `promote` job uses `release-bot` (automatic, every push to main). scripts/ci_preflight.py now refuses any job that mints the release App token without a literal environment; the tests prove the refusal and its controls. The jobs keep reading the same secret names, so they work before and after the owner copies the secrets into the environments.

Not to be merged until the owner confirms the two environments and their secrets exist.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…native capture tool and a drift test

Foreign readers of the report-sink stream (a worker-side TypeScript reader, job
rows) cannot run docker/pf_sink.py. docker/gen_sink_vectors.py writes one
versioned set of byte-exact streams. The Python vectors are produced by the
Python test fixtures that mirror the writer's framing, not captured from the C++
writer: a complete stream with a record of every kind (E, T, O, V, K, X, H) and
the trailer, two streams with a bad trailer (counts, sha256) and two truncated
streams (inside a record, at a line boundary), each with the expected result
recorded from the engine's own reader (the parse as JSON, the converter's
output, or the exact exception class and message). Their header carries
placeholders (empty run_id, engine "test"); the page lists the header fields a
real stream fills. docker/capture_native_sink_vector.py runs docker/run_json.py
on a built hand-written strategy library and a small bar file and saves the
stream exactly as the C++ writer wrote it, with the reader's parse and a
provenance record; the generator's --native-dir adds those files to the
manifest, which carries the wire version, the engine commit, the generator
command and every file's size and sha256.

tests/data/sink_vectors_v1/ holds the set. docker/run_json_sink_vectors_test.py
regenerates the Python vectors and compares them byte for byte with the
committed files, re-reads every committed stream with the reader and the
converter, and for the native files checks the manifest hashes, the reader's
parse and a real header. docs/pages/sink-vectors.md describes the files, what
the set does not cover, and how a foreign reader uses them.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Bring the branch up to date with main so its checks run on the current tree.
No conflicts; no file of the branch changed by the merge itself.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ith a doc-lint rule that holds it

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… tree builder reads

CiFileLoading.make_tree copies the actionlint config and tests/CMakeLists.txt
using self.CONFIG and self.CMAKE. test_second_workflow_files_are_discovered_by_the_ci_entrypoint
lends it an AppTokenEnvironments test that had neither attribute, so the
preflight-tests stage failed with AttributeError before any assertion ran.
The test now carries the same two class constants; no assertion changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Bring the branch up to date with main so its checks run on the current tree.
No conflicts; no file of the branch changed by the merge itself.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…in the number of bars

Strings and maps now live in the paged object store instead of the C++
heap, so their memory does not grow with the number of bars a script runs.

- ScriptString and the map image keep their bytes in store objects; split,
  concat and the map operations read and write through the store.
- Store transactions use a fixed pool of handles (four caller slots plus one
  cleanup slot). Running out throws TxPoolExhausted before any state
  changes; there is no heap fallback. Store::scope() joins an active
  transaction without taking a slot.
- The join hooks join_active_tx() and abandon_joined_tx() are public pure
  virtuals so forwarding wrappers can pass them on. begin_cleanup() is used
  only by the map destructors, and note_cleanup_skipped(CleanupSkip) records
  whether a cleanup was skipped because the transaction was rollback-only or
  because it failed.
- A store refuses the (limit+1)-th live object with
  scratch_capacity_exceeded. The limit is Shape::max_live_objects, default
  and ceiling 1,048,576, counted per store and rolled back with the
  transaction. The error reuses the cohort resource name for now; a
  dedicated name follows before the 1.8.0 tag.
- Tests cover string and map parity, rollback, nested destroy, split, the
  pool limits, the live-object cap and wrapper forwarding, plus 100k and 1m
  growth runs for strings, maps and prepared maps.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge; keeps the strict ruleset's up-to-date requirement.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Two conflicts, both in the source gate. The scanner change here retires the entry for the
trade-ledger iterator index (it no longer matches a site), so that entry stays removed while main's
new entry for the command scratch cache budget is kept. Main also added two entries that reuse the
ids this branch had given to the object-limit message and the map iterator temporary; those two
move to U1545 and U1546. The unreviewed ceiling becomes 1281 (1279 here plus main's 2), the
pending-flat-memory ceiling stays 185, and the pinned OPEN count becomes 1466, the number of
entries (1463 on the common base, plus 2 from main, plus 23 added and minus 22 removed here).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… drop the stale 180 s wording

test_selected_window_ownership_manifest_controls_shard_1of4 took 74 s on an idle
8-vCPU box while shards 2-4 took 41-50 s (the modulo split over sorted case ids
put the heaviest controls together). The four rows carry no explicit TIMEOUT
any more (#504 removed it) but the tree still described a 180 s budget and the
CMake guard still exempted the exact old statement.

- Shards are now assigned largest-first to the lightest shard, with a weight per
  case read from its AST (checker subprocess launches, through self helpers and
  literal loops). No measured seconds are stored. The partition stays exact,
  order independent and deterministic; --shard I/N validation and the
  "ran N tests as selected" fail-closed check are unchanged.
- The two 60 s subprocess deadlines inside the controls are removed.
- scripts/check_cr150_cmake_d.py no longer exempts the old six-row
  "TIMEOUT 180" statement, and its test now requires the refusal.
- docs/ci.md, tests/ownership/README.md and the CMake comment describe the
  balancing and no deadline.
- New rows: balance bound (max/min <= 5/4, violated by the old split) and an
  estimator row; the partition row now checks input-order independence.

Documentation sentences removed from docs/ci.md, all introduced by 375795a (the
change that split the ownership manifest controls into four CTest shards): the
sentence saying that change splits the controls into four CTest shards
(`test_selected_window_ownership_manifest_controls_shard_1of4` through the
fourth), the sentence saying it adds three public registered/run rows to every
engine profile in both fixture worlds and only raises the counts above the
pinned minimum floors, and the sentence saying the private-fixture and receipt
row populations stay the same. The replacement paragraph states the balanced
four-shard split and the unchanged row populations.

Measured on a throwaway 8-vCPU spot VM: shards 51/57/53/55 s after, 74/42/41/50 s
before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…er-platform stream memory bound

The shared descriptor guard compares every status and descriptor flag at each
boundary and masks only Darwin's kernel-maintained "descriptor was written"
bit (FWASWRITTEN) under __APPLE__. A new test plants a real F_SETFL, F_SETFD
and close after a pipe write and requires the guard to reject each; two more
ctest rows run those planted modes through a small CMake driver that accepts
only exit status 1 together with the guard's own message.

The 64 MiB stream memory test keeps its 1 MiB bound everywhere except Darwin,
where it allows 2 MiB for the 16 KiB page size.

CHANGELOG notes the macOS report-sink limit is no longer listed (re-examined,
not re-measured). 1.8.0 is Linux only, so no CI job exercises the Darwin mask
or the Darwin bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Attribute exclusive opens and fork-descendant scratch files to their creating
process, keep unrelated co-tenants out of the witness, and report the offending
path with its PID or an explicit unattributable reason. Add three public Linux
CTest cases for exclusive open, inherited evidence, and diagnostics.

Carry the three rows through verifier floors and mock accounting. Preserve
documentation line counts and refresh the floor-table citations.

Documentation replacements from 3ea5a0d name the old wording:
"Their default runs now enter strict exclusion accounting"
"Verifier 8 + preflight 8 + command-after 8 + report 4 add 28 public rows; plain Debug-type runs enforce the increased registration floors."
"Release 938 with the overlay or 631 without it"
"The floor itself remains fixed at 631/334 absent or 938/365 present"
"The native PR registration floor is 1000"
"Both numbers are constants of the profile"
"The full-run release and kernel floors are the table's own numbers above"
"EXCLUDED_REGISTERED_MIN holds 904 for debug, 895 for sanitizers"
"The run floors in the table above"
The release and kernel table rows from 3ea5a0d use the new floors.

The population wording and table header from b97c409 are replaced:
"The registered column is why the absent floor is not the absent registration, and it is to be re-measured after the registration fix lands."
"registered (present / absent)"

The four citation edits retain the passages named by:
"The floors' numbers live in one place"
"Each profile counts the rows that"
"the minimum number of CTest rows"
"The profile carries a row floor"

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge; keeps the strict ruleset's up-to-date requirement.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…inal docs wrap

The App-token environment guard recognised only actions/create-github-app-token
by action name; a tibdex/github-app-token step fed from vars, inputs or a secret
whose name does not say "private key" passed without an environment. Both
actions are now recognised by name, with a refusal row and a removal control for
every ref and quoting form and every key source.

The docs paragraph about the deployment environments is shortened to its facts
(no internal card id) and the four neighbouring sentences keep their original
line wrapping, so the paragraph is a pure addition to docs/ci.md.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ed reviewer

Read back from the repository settings: both environments allow deployments only
from main and neither has a required reviewer, so the docs sentence no longer
claims an owner review for `release`.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Corrects four comments that no longer match the code they describe, and one
stale line number in a script comment:

- scripts/ci_verify.py: the selected-sink memory test is registered at its
  current line of tests/CMakeLists.txt.
- tests/test_scratch_create_pinned_text.cpp: the header names the files that
  produce the pinned create diagnostics instead of a release.
- src/source/pine_adapter.cpp: the placement table scans the retained
  placements and consults the historical minima for erased rows; it has no
  per-id index.
- include/pineforge/native_host.hpp: the live-order read comment no longer
  claims the reads allocate nothing, since a spilled row is read into a buffer.

No code changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… path it pins

The header named two producing files; the test also pins the id-pair scratch
file and the successor-roots create. It now lists all four scratch paths by
what they are, and drops a stale note about an unobserved prediction. Comment
text only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…labels

Reword published documentation that carried internal task and card labels so each
passage says what the thing is. Lines that only changed a label keep their meaning.

Sentences replaced or dropped, named by the commit that introduced each:
- 91353de: "Read-only census for CR-150 L22u / A6 and A5-WIDER." and
  "All eight vectors use the three-term verdict;" (arena-bound pages), and the two
  passages of cr150-contracts.md that cite the external design lane's work files
  and its revision 6 amendment.
- d093560: the paragraph of native-refactor-progress.md that quotes the supervisor
  cap and the decision that stopped the R5 performance wave.
- a77fb73: "R4-D L1 advanced BacktestEngine, NativeStrategyHost, and the private
  consumer to engine_script_run_v17;" (abi-stability.md).
- 44bb687: "R5 L4 adds NativeRunSpec::margin, folded only when it is set;" and
  "R5 L5 adds calculation, max_recalculations_per_point and open_ba..." (abi-stability.md).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ds instead of internal tracking labels

Replace internal task, card and work-lane labels in source comments, docstrings,
workflow comments and human-readable progress lines of test programs with plain
descriptions. About 620 text lines in 285 files; no identifier, file name, CTest
selector, CMake fence, assertion or data file is renamed or edited.

Files whose exact bytes are pinned or whose text is data stay unchanged: the
frozen run_json fixture (its digest is asserted), fixture sources hashed by the
heap census, the ci/ gate data, the broker-state waiver list, the ownership probe
notes, the heap census report heading and the run-failure label passed to
note_run_failure. Printed lines that changed are listed in the pull request; each
old text was searched across the engine and the other repositories and nothing
reads them.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…table bytes

check_kernel_residuals.py fed the whole debug-stripped kernel archive to
`strings -a`, so instruction bytes in a text section could spell a forbidden
name by chance (four code bytes were reported on a macOS Release object). The
`strings` surface is now built per archive member from section payloads:
executable sections, sections without file bytes and an explicit list of
debug/linker metadata sections are skipped, every other section (including
unfamiliar names) is scanned. `nm -C` still reads the whole archive.

Duplicate member names in the archive are an InfrastructureError, because
`ar p <name>` cannot select a later member with the same name. A truncated or
unrecognised object structure is an InfrastructureError as well; there is no
fallback to scanning the whole object and no object or offset exception.

Tests: hand-built ELF and Mach-O byte fixtures (thin, fat and fat64, 32 and
64 bit, both byte orders, ELF extended section counts) show that planted code
bytes are ignored while the same word in a data section, an unknown section or
a symbol is still flagged. The macOS fixtures are synthetic, not compiler
output; 1.8.0 is Linux only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…aragraph adds

Two preflight tests pin docs/ci.md to an exact line count (the post-fleet docs
test and the documented slow-row count test). The environment paragraph is now a
pure addition of one blank and one paragraph line, so both pins read 2016 instead
of 2014. Nothing else in either test changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ps no longer overflow int

The key that buckets equity points into months, (year * 12 + month), was
computed in int. For timestamps far from the epoch (INT64_MIN milliseconds is
about year -292 million) the product overflowed, which is undefined behaviour,
and the libc fallback for seconds beyond +-2^40 could fail and leave an
unfilled calendar. The key is now std::int64_t end to end: the UTC helper, the
memo, the monthly walk and the fold in metrics_fold.hpp. The UTC key always
uses the civil-date arithmetic, which is exact over the whole millisecond
range, and never calls gmtime_r; the chart-timezone key widens the year before
it adds 1900 and multiplies. A local-time lookup that libc cannot answer now
raises the existing report_result_invalid refusal instead of returning a key
read from a zeroed structure. Keys for in-range timestamps are unchanged.

tests/test_utc_month_key_arithmetic.cpp checks the key against an independent
Gregorian day count written in the test, at INT64_MIN, INT64_MIN + 1,
+-2^41 s, INT64_MAX, around the arithmetic span edges, and over a sweep of
month boundaries and leap days.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The two-line comment above the EXPORT_NAME property moved the line from 173 to 174.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 and others added 25 commits October 28, 2026 09:21
…y rows to the one-pass seam, sync the report artifact once

Three changes on top of the one-pass report assembly, all in the Python wrapper and its tests.

Fingerprint block. The bulk bar-loader constants (`_BAR_HEADER`, `_BAR_FLOAT`,
`_BAR_CANONICAL_LINE`, `_BAR_BULK_SLAB_ROWS`) sat between the
`# >>> fingerprint helpers` and `# <<< fingerprint helpers` markers, which must
stay byte-identical to the same block in `scripts/run_strategy.py`. They now sit
below the end marker, defined before their first use; the two blocks match again.

Bulk-equity rows. Eighteen test groups in `docker/run_json_bulk_equity_test.py`
and `docker/run_json_bulk_equity_fuzz_test.py` (246 failing rows) watched the
separate bulk pass that the one-pass assembly removed. Each now observes the
same property through the one-pass seam and compares against the frozen copy of
the earlier reader in `tests/fixtures/run_json_frozen_6b33c47f_rjone.py`: byte
equality on every window edge, tampering after verification, strict refusal type,
text, code and priority, and the memory and flatness rows. Four test functions
are renamed, none dropped, no assertion loosened. The product behaves like the
frozen reader on all of them, so no product behaviour changes here.

Report artifact durability. `_BoundedJSONFile._sync_window` wrote, truncated and
synchronised every full 64 KiB window. It still writes each full window before
reuse and still truncates the padding of a last partial block before any read;
the data synchronisation now happens once in `flush`, before the artifact is
read back or published, and only when bytes were written since the last one.
Short-write, EINTR and errno text are unchanged, and a failed synchronisation
is still raised by the same call. New rows in `docker/run_json_fused_test.py`
count the synchronisations, fail the flush, and compare the bytes with the
frozen reader.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…e one-pass change

The one-pass report assembly and the follow-up commit moved code in
`docker/run_json.py` and `docker/pf_sink.py`, so 149 `file:line` citations in
`docs/design/BARS-BIN-INPUT.md`, `docs/design/selected-window-wire-v1.3-revision-3.1.md`
and `docs/pages/memory-management.md` pointed at the wrong lines. Each citation
now names the lines that hold the symbol the sentence talks about. Only the
numbers change: no doc gains or loses a line, no citation is removed, and no
sentence is reworded.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A hosted admission workflow (gke-pr.yml, the only pull_request_target file)
admits same-repository pull requests from members and runs the PR proof legs
on bare gke-<size>-<arch> pods from the immutable admitted head; ci.yml keeps
authz only. The admission preflight is repaired as follows.

Derive the admission linter command from its existing single-prologue
constant, restore ci.yml's canon authz job verbatim, point the fixture-floor
documentation test at the full-profile Verify workflow, and reject embedded
bare CR characters while accepting normal CRLF. Add specific regression and
mutation controls without changing the reported parser-differential row.

Convert only the four gke-pr.yml runs-on blocks to bare scale-set scalars,
keeping their existing 8/32/16/8 CPU sizes and amd64 architecture. Both
admission and general runner checks pin the raw scalar; the refusal tests
cover mappings, arrays, self-hosted arrays, expressions and other wrong
scalar forms. Other workflows retain their current pins.

docs/ci.md replaces the sentence introduced by 0212cab because admission now
classifies the immutable compare list before scheduling proof. Its replacing
sentence is "Preflight and the proof jobs start as soon after admission has
classified the compare list and chosen the verifier tier."

docs/ci.md replaces the twelve sentences introduced by 065ef1d because cache
restore/save and optional cold-mode guidance no longer describe these jobs.
The replacing sentences are "Every trusted post and PR proof job is a cold
proof by construction." and "Post profiles build cold, so their results do
not depend on objects written by a pull request." The earlier cache-log and
ASan diagnoses are historical; this change establishes no runtime fix.

docs/ci.md replaces the four sentences introduced by 0983770 because the
workflow compiler cache is now local to a cold job. The replacing sentence
is "Pass --ccache when ccache is installed; it is an empty local directory
on the GKE proof and trusted post jobs."

docs/ci.md replaces the sentence introduced by 0d76a09 with current registry
ownership rather than claiming a new timing measurement. The replacing
sentence is "The registered rows remain pinned in tests/CMakeLists.txt; the
post profiles execute their full population on the verification hosts."

docs/ci.md replaces the four sentences introduced by 133f571 because PR
classification now uses the payload-bound compare list and a second read,
not a first-parent checkout diff. The replacing sentence is "The base tree's
copy judges the list, so a PR cannot change the classifier that admits it."
The replacing timing cell says "Seconds after the second compare read and
job-result checks."

docs/ci.md replaces the twelve sentences introduced by 21ba62b because
admission, preflight and the hosted result now own the former changes/build
aggregate policy. The replacing sentences are "After successful admission,
preflight always runs for an allowed PR." and "The hosted result job
requires admission and preflight success, accepts a skipped proof group
only for the two classified classes, and applies a one-way code-extension
veto."

docs/ci.md replaces the twenty-six sentences introduced by 3a0d150 because
PR proof work moved to the main-defined admission workflow and both PR and
trusted post jobs now build cold. The replacing sentences are "The only job
of ci.yml is authz; PR proof work lives in gke-pr.", "PR proof legs build
cold; trusted post workflows also build cold and contain no Actions cache
restore or save.", and "gke-pr inlines the corpus subset for ordinary code
PRs and skips it only after the base classifier proves docs-only or
GitHub-only." The opening trigger table and hosted result replace the
former PR matrix and build-gate descriptions.

docs/ci.md replaces the two sentences introduced by 3ea5a0d because shared
Actions compiler-cache keys and shard-1 publication were removed. The
replacing sentences are "Native diagnostic artifacts carry the shard index;
compiler and curl installations have no Actions cache." and "Trusted post
workflows and PR proof legs build cold; no trusted job restores an Actions
cache."

docs/ci.md replaces the two table entries introduced by 6c95d82 because the
opening table now separates admission from full post verification. The
replacing ci-post.yml proof-jobs row says "Complete Release, Debug, native,
kernel and parity profiles"; the later full-check and arm64 paragraphs
retain the architecture detail.

docs/ci.md replaces the three sentences introduced by 91353de with the
admission/post trigger map, while retaining the later precise exclusion
clarification: "The verifier's Debug-type default excludes slow from Debug
and sanitizer runs on every event, including non-PR events; Release, kernel
and native retain every row." Complete profiles still mean the verifier's
current profile contract, including those Debug-type defaults.

docs/ci.md replaces the table entry introduced by b0f4341 because ci.yml
now retains only hosted authz. Its replacing ci.yml row says "authz:
hosted ubuntu-24.04; no proof legs", with gke-pr.yml listed separately for
hosted admission/result and the GKE proof jobs.

docs/ci.md replaces the two sentences introduced by c85a40b because
main-defined pull_request_target admission is distinct from direct
pull_request routing. The replacing routing sentence says "Verification
jobs keep literal runner pins at the workflow boundary: GKE PR admit is
hosted, while PR preflight, proof legs and parity use gke-runners;
credentialed post work keeps its protected fleet." The event refusal now
says "pull_request workflows cannot name GKE routing." The protected
runner group still selects the admission scale sets, now named by bare
scalars.

docs/ci.md replaces the sentence introduced by d093560 because the subset
fetches and verifies its feed directly instead of restoring corpus/data.
The replacing sentence is "The subset job checks the submodule out with
GIT_LFS_SKIP_SMUDGE=1, fetches the ~176 MB feed directly and checks every
byte against the sha256 in the corpus's own Git-LFS pointer before running
the probes."

docs/ci.md replaces the three sentences introduced by eed6598 because the
hosted result now judges the admitted proof group rather than the former
selected build matrix. The replacing sentences are "The hosted result job
requires admission and preflight success, accepts a skipped proof group
only for the two classified classes, and applies a one-way code-extension
veto." and "Failed, cancelled, missing or unexpected jobs remain red."

docs/ci.md replaces the sentence introduced by 482b000 ("PR build/preflight
and parity subset use the pf-gcp-pr fleet; post preflight/parity use bare GKE
scale-set names; ...") because PR proof legs now run only in the admission
workflow on bare gke-<size>-<arch> scale-set names and ci.yml keeps authz
alone. The replacing sentences are in the opening trigger table and the
"Verification jobs keep literal runner pins" paragraph.

The quoted-key expectation in test_harmless_spellings_are_not_findings visits
ci-post.yml before docs.yml, the order the checker reports them in.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… a site

The reviewed site inventory still listed the assignment of index_ in the
cohort view iterator of native_order.hpp. On the combined tree the scanner
no longer reports that site, so the source-bounds gate stopped on it as
stale. The scope text of that record was last rewritten for the
native_order_v8 namespace rename (aab50fb); the gate passes on that
commit alone and fails only on the combination.

Remove that one record. The gate then reports no findings and its own
tests pass. The scanner's emission for this tree differs from the committed
inventory only in the spacing of one spelling (std::string in
src/detail/o1_store.cpp), which the gate does not distinguish; that record
is left as it is.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Run the complete preflight test module in up to eight CPU-sized shards,
and explicitly independent stages in a four-worker bounded pool.
Keep stage inventory, ordered summaries, exit semantics and separate logs.

docs/ci.md replaces the sentence beginning "A failure fails the advisory
aggregate build job and retains logs in build-ci-preflight/" with the
per-stage log and declaration-order summary description. The page keeps
exactly 2014 lines.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…med tests

This commit changes no file. It corrects two statements in the message of
801e4e6 so the squash message can be written from accurate text.

Tampering after verification. 801e4e6 says the product behaves like the
frozen reader on all eighteen rewired groups. That is wrong for one group. The
earlier wrapper read the sink a second time and refused when the file had
changed after the first read ("the report sink changed after verification").
The one-pass wrapper reads the sink once, and the report it publishes uses
exactly the bytes hashed in that read, so a sink rewritten after that read does
not change the published report. The second read that refused is gone by
design. The group, now `test_tampered_source_scan_and_fused_artifacts_match_frozen`,
pins the one-pass behaviour; the tests are right, the sentence was not. Anyone
who relied on the refusal after a rewrite should know it no longer exists.

Renamed tests. Four test functions were renamed in 801e4e6, none dropped:
- `test_tampering_after_verification_ends_like_the_strict_path` is now
  `test_tampered_source_scan_and_fused_artifacts_match_frozen`
- `test_copy_guards_and_completed_prefix_match_strict` is now
  `test_sink_scan_guards_and_fused_refusals_match_frozen`
- `test_bulk_passes_use_flat_python_memory` is now
  `test_fused_bulk_and_artifact_readback_use_flat_python_memory`
- `test_control_a_regex_call_over_a_whole_window_fails_the_memory_check` is now
  `test_unbounded_regex_breaks_fused_memory_limits`

Retained reader path. `_ReportSink` always runs the fused pass, so
`_verify_legacy`, the legacy branch of `_build_sink_report` and
`_stream_equity_array` cannot be reached from `main()`. They stay as a
reference reader that the bulk-equity tests (`_scan` in
`docker/run_json_bulk_equity_test.py`, and the fuzz file) compare against; those
tests are its only callers.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Main now carries the squashed first five members of the previous stack.
The only conflict was in CHANGELOG.md: keep main's lines and this branch's
own macOS report-sink note, which sits between the walk-forward entry and
the supported-platforms entry. The merged CHANGELOG.md equals this branch's
file; nothing else needed a hand edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit, including CHANGELOG.md.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit, including CHANGELOG.md.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit. The branch also carries the one-record fix to the reviewed
source-site inventory (the record for the cohort view iterator's index member
in native_order.hpp, which the scanner no longer reports).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit, including CHANGELOG.md. The run id line in docker/entrypoint.sh
is byte for byte the member's.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. Two conflicts,
both resolved by keeping the member's side of the lines it rewrote:

- docs/ci.md: the opening sentence of the trigger map is the member's one
  sentence; the paragraph it replaces is the one main reworded.
- scripts/ci_preflight.py: the member's gke-pr.yml checks, trusted cache
  checks and authz-only check replace the pull-request pipeline check call
  (the member removes it), and the app token environment check from the
  promotion-guard member stays, first, exactly once.

The resolved ci_preflight.py differs from the member's file by exactly the
promotion-guard member's own 80 changed lines.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. One conflict, in
docs/ci.md: the paragraph that describes the preflight pool and where the
proof legs start. Resolved with the text CI's reviewed combination of the
admission and parallel preflight members carries for that paragraph, taken
verbatim.

scripts/ci_preflight.py and scripts/test_ci_preflight.py merged without a
conflict; they differ from that combination only by the earlier members'
own lines.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…fault cap, so reading every closed trade each bar no longer re-reads the scratch files

A run with a report sink installed gave the trade-ledger read cache 128 rows. The cache is direct-mapped, so a strategy that reads strategy.closedtrades.* for every closed trade on every bar (a cyclic scan of N rows with N well above 128) missed on nearly every call, and each miss costs a positional read of the 168-byte record plus up to four for its strings, from files whose page cache is dropped every 64 KiB. Such a run did not finish.

kSinkCacheRows is now the default 65,536 rows. The declared bound is constant and independent of the trade count: 65,536 x 312 = 20,447,232 entry bytes plus sizeof(TradeLedger) + sizeof(State), allocated lazily (powers of two up to the cap), so a run with few trades still holds few entries. The cache stays read-through and is invalidated exactly as before: no returned value, report byte or hash changes.

Doc sentences replaced on the memory-management page, named for the doc-revert check (introduced by 6d2bf95): "With a report sink installed: 128 entries (`kSinkCacheRows`)."; the 312 B per entry sentence that gave the old 39,936-byte sink figure; "Chosen by the engine, not settable from the C API".

Tests: test_ledger_sink_scan runs a sink run with 2,000 closed trades, reads profit and entry id of every trade on six per-bar scans, counts every positional read, and compares the values and the serialized report with a second sink run whose ledger cache is forced to 128 rows (the two report files must be byte-identical). test_p2cx_tledger_budget now asserts the exact 20,447,232-byte bound and the full resident_cache_bytes() figure. The sink heap-flatness witness (test_selected_metrics_l17b) accounts for the cache cap separately from the two scratch windows and checks that heap use is flat between 70,000 and 100,000 trades. The memory-management page states the new bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…the sink fd test

The page cites the line where the sink fd test freezes time.time. The run id
forwarding change added four lines above it in that test, so the line is now
1271, not 1267, and the doc-anchors stage reported the stale citation when the
stack was combined. Only the number in docs/pages/sink-vectors.md changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the reviewed head of #611, which already contains #609, so that the one-pass report change is verified on the tree it will land on. No conflict resolution and no other change.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Comment thread docker/gen_sink_vectors.py Fixed
Comment thread docker/run_json_fused_test.py Fixed
luisleo526 and others added 2 commits October 28, 2026 16:16
…after the merge

Eighteen `docker/run_json.py` citations in `docs/pages/public-contract.md` pointed at lines that moved when the one-pass report change and the report-cache change were merged. Each citation now names the line that holds the construct its sentence describes: the selected-run helpers, the capability probe, the report writer, and one `add_argument` line per command-line flag. Only the numbers change; the page keeps its line count and no sentence is reworded.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Main now holds the squash of the stack this branch already contained; the merge changes no file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@luisleo526
luisleo526 merged commit 1654497 into main Oct 28, 2026
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant