Repository navigation
v1.11.1 engine train: #797, #801, #789 - #808
luisleo526 wants to merge 29 commits into
Conversation
Exercise the version-only reset before tagging, preserving tagged behavior. Add refusal controls for parent count, extra paths, version direction, catalog bytes, existing tags and inherited release stamp constraints.
Require each release phase to use its own exact-commit CI evidence. Keep the bump untagged until its checks finish, then authenticate and tag that same commit. Validate the two-file transition and preserve catalog assets.
Record advisory verification and complete workflow attempt history, bind the decision to received response bytes, and refuse changed run metadata before admission. Keep release receipts on refusal and honor declared catalog paths.
Use timezone-aware UTC and keep exact output assertions aligned with each test input. Preserve every release safety condition and mutation control.
Cover authenticated advisory creators and first-attempt actor refusals. Reject malformed advisory states before logging them, retain sanitized unexpected-exception receipts, and distinguish invalid triggering actors from changed attempt metadata. Keep parity as the only required status.
Carry both strict admission forms in the hosted release workflow, retaining caller identity, artifact custody and exact-commit release evidence.
Keep historical matrix inputs independent from the phased release workflow and require both phase and dry-run boundaries on downstream publishing jobs.
Choose the expected guard and message from the complete dispatch declaration, while retaining strict workflow acceptance and both dry-run boundaries.
…hosted jobs WorkflowPinTests reads the #768 matrix rebuild of release.yml for its hosted-matrix contracts. That rebuild keeps the shipped two-phase release job, a combination the validator correctly refuses, and its single prebuilt job has no tag-phase guard. Four checks therefore failed. They now read the shipped workflow: - test_two_release_phases_stop_before_publication_until_tag_phase pins the exact job-level tag-phase if: of prebuilt-x86, prebuilt-arm and upload, and the closed job list. The notify-hub, gate, step-condition and writer pins are unchanged. - test_prebuilt_requires_release_success_without_failure_bypasses holds the release-success pin for the release job, both build jobs and the publishing upload job: needs: release on each build job, upload's exact needs: [release, prebuilt-x86, prebuilt-arm], no always(), !cancelled() or continue-on-error: outside comments, and no release_bump_workflow_findings. - test_prebuilt_failure_bypass_mutations_are_refused, both subtests, inserts each bypass into each build job and into upload and requires the same refusal. Each of those jobs first passes with the shipped "no continue-on-error" comment in it, so the pin is still shown to read keys, not words. The validator compares every split job header, so the pin now scans the build jobs and upload before the release job, and a refusal names its job and bypass. The other WorkflowPinTests tests keep reading the matrix rebuild. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scripts that read their closed trades again and again (strategy.closedtrades accessors in a loop) were about 1.9 to 2.0 times slower than on 1.10.0 on histories of 15,500 and 39,456 closed trades. Two things made them slow: every cache hit rewrote the rank of the other entries of its set, and four-way sets spread consecutive trades over four times the pages their rows need. The cache now follows the number of closed trades. While the ledger holds at most as many rows as the cache has entries (215,070 by default), row n owns slot n: one way, one probe per lookup, and only the pages the rows reach are committed (25,000 closed trades hold 98 pages, 7.8 MB, where four-way sets held 391 pages, 31.2 MB). With more rows the four-way sets apply as before. An append that takes the ledger past the cache's rows, or a rewind back to them, drops every cached row and keeps the allocated pages. In a set, a hit on the most recent entry changes nothing and a hit on any other entry ages only the entries more recent than it, so a full set replaces the entry touched longest ago, as the documentation says. A miss is probed and counted once. Every value a script reads is unchanged: the reports of a 15,500-trade script are identical field by field. Measured on an aarch64 build host, one pinned core, engine time against 1.10.0 built with the same compiler: a script that scans 15,500 closed trades again and again takes 1.03 of the 1.10.0 time (five rounds; main before this change 1.92); a script that only opens and closes trades is unchanged by this change (1.10 on 150,000 bars, as on main). With 250,000 closed trades, above the cache's rows, a script that scans all of them every 6,000 bars takes 40 s, against 27 s on main before this change and 138 s on 1.10.0 (two rounds). Timings on x86-64 were not measured. Tests: the placement oracle in tests/ledger_default_cache_rows.hpp now takes the number of rows the ledger holds, and nine tests that compared resident bytes or entry counts with it state both placements; three new tests pin the least-recently-used rule, the single probe of a miss and the placement change. Documentation: docs/pages/memory-management.md and the header comment describe the placement by row count. Two sentences of the memory page leave in the ledger read cache row: the sentence that began "Storage is private: a global ordinal chooses one four-way set" (introduced by ba14721) is rewritten to state the placement for both populations, and "Page growth preserves entries through the 64/128/256 boundaries." (introduced by 69cddc9) now says that growth preserves entries through every page boundary, every 256 rows within the cache's rows. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Preserve phase-aware release admission while incorporating the current main toolchain contract. Keep both Eigen flag constants and the mode parameter, and update the planned-phase fixture to the typed PATH flag. Co-Authored-By: Codex <[email protected]>
Retain the tagged VERSION and catalog baseline before exact-head validation. Co-Authored-By: Codex <[email protected]>
Complete the selected-capacity production entry and add pineforge-selected-admit, a small command-line program that checks a caller's capacity bounds on the Source and the selected inputs of a run before the first writer consumes them. Production entry: the five selected-capacity observers are bound through the native materializer, and actual Source records and CSV bytes are accounted with the existing reserve primitive. Input-sensitive tests cover the field limits, the exact failure arguments, the byte accounting and the production binding. Helper: it reads a versioned request descriptor (tools/selected_admit/ DESCRIPTOR-V1.md specifies the request, the result and the manifest), takes custody of each input by descriptor number, size and digest, parses it with the existing selected-capacity reader, calls the product entry, and answers with one result document and an exit status. The Source stage forwards each record only after it is admitted, and refuses a forward descriptor that is the same file as its stdout or stderr (exit 41); the Selected stage counts the selected pass and retains nothing. The helper and its manifest (size, sha256, build identity, schema digests, stages) install to bin/ and share/pineforge/. The sha256 of the spec is compiled into the manifest, so a change to the spec changes the manifest. The spec also lists what the Source stage holds in memory (the record window and the reader's copy of one field, each up to the record cap, a chunk and the process base) with one measured figure, and what the helper reads from the environment and the file system: GLIBCXX_TUNABLES at start, and, for a calendar zone that is not UTC-like, TZ and, per libc, TZDIR and the zone file under the zone database root. Tests: descriptor, custody, stage and manifest tests, the Source-stage parity test, an import-list test and a calendar test. The import-list test reads the undefined symbols of the built helper with nm and fails if it imports one of its list of 41 process, thread, session, socket, descriptor-duplication, signalling, system-call or library-loading functions, other than dlopen (one RTLD_NOLOAD lookup of the engine's own image) and recvmsg (the run-quota endpoint). The two allowed names are pinned in the test, a glibc fortified or internal spelling of a forbidden name counts as the name, an nm line that cannot be parsed stops it, and a listing without pread, poll and __cxa_begin_catch, which the helper builds measured so far import, is rejected as empty, truncated or unrelated; having them proves neither completeness nor which file nm read. The calendar test runs the Selected stage with New York four-hour script bars across the two clock changes of 2024: the open of the date's own season is admitted, the open of the other season is refused with the neighbouring bar opens that civil-time arithmetic gives, and a zone database root that holds no zone makes the zone unsupported. Inventory: the allocation-site inventory gets the three new Source-stage groups and five site rows, the unreviewed ceiling goes from 1410 to 1413 and the exact-union pin to 1356 + 57 + 184, the five selected limit routes of the ownership manifest are corrected, and the CMakeLists.txt line citations in five documents are re-anchored (three citations that named a neighbouring line now name the line they describe). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The capture counter starts after the ledger cache has been populated, so resident cache-page growth must not be required in its working-heap peak. Restore the symmetric fixed-window comparison while retaining the exact row-count-dependent cache residency assertions and all output oracles.
Retain the row-count-derived page-growth upper envelope, since Release can allocate cache pages during capture. Do not require that growth in the measured peak when pages are already resident. Keep the exact cache residency, scratch-window, output and profile checks unchanged.
…utral names The registration file of the helper's unit tests, the fence around its script tests in tests/CMakeLists.txt, and the CTest label of all seven tests carried a name derived from a work card. They are now selected-admit-helper and selected_admit_helper. The registrations are otherwise unchanged, and nothing else in the tree named the old file or label. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The private-fixture reads check reads the add_test arguments of tests/CMakeLists.txt and refuses a variable that starts with _pf_ unless it saw that variable set in the files it reads. The manifest test took its expected stages from _pf_sa_stages_json, which the root CMakeLists.txt sets, so the check stopped on it and test_private_fixture_reads and its controls failed in every build that runs the full suite. The pull request always carries tools/selected_admit/stage_source.cpp, so the root sets that variable to ["source","selected"]; the test now gets exactly that text, and it still compares it with the stages in the built helper's manifest. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…rule CMakeLists.txt takes the sha256 of tools/selected_admit/DESCRIPTOR-V1.md into the selected-admit helper's manifest at configure time, so a change to that page changes a build output. The docs-only rule called every Markdown file outside its excluded trees documentation, and test_no_cmake_file_reads_documentation (the docs-only-tests stage of preflight) failed on exactly this page: Lists differ: ['tools/selected_admit/DESCRIPTOR-V1.md'] != []. The page joins PROOF_READS, the list of documentation inputs that proof jobs read, so a change to it runs every proof leg. The pinned list in the rule's test, the scan test that now requires the page among the CMake reads it must see, and the sentence in docs/ci.md that counts the excluded files are updated with it. A new test replays the page in real git histories: an edit, a rename away, a rename onto its path and a deletion all run the proof legs, and another Markdown page of the same directory stays documentation. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ision evidence docs/ci.md: replaces the release-bump exemption sentence introduced by 72d9fd5 (#486), 'When there is no qualifying merged PR for the current main sha, phase 1 permits one release bump only when all three conditions hold:', with the phased release-bump and tag-admission wording of #789. Co-Authored-By: Claude Opus 5.5 <[email protected]>
| 'release.yml job prebuilt-x86 must not receive the protected release ' | ||
| 'environment or release credentials', |
| 'release.yml must keep the exact two-phase release writer, ' | ||
| 'untagged bump push and same-SHA tag retry', |
| 'release.yml must keep the unconditional release gate with its release ' | ||
| 'App token for the App-only main pull_request barrier', |
| 'release.yml must always retain the exact release gate receipt artifact ' | ||
| 'before planning or publishing', |
| 'release.yml job prebuilt-x86 must not receive the protected release ' | ||
| 'environment or release credentials', |
| 'release.yml must keep the exact two-phase release writer, ' | ||
| 'untagged bump push and same-SHA tag retry', |
| 'release.yml must keep the unconditional release gate with its release ' | ||
| 'App token for the App-only main pull_request barrier', |
| 'release.yml must always retain the exact release gate receipt artifact ' | ||
| 'before planning or publishing', |
| 'release.yml must keep the exact two-phase release writer, ' | ||
| 'untagged bump push and same-SHA tag retry', |
| 'release.yml must keep the unconditional release gate ' | ||
| 'with its release App token for the App-only main pull_request barrier', |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Splits the release workflow into two dispatch phases: bump gates the dispatch SHA and pushes only the untagged two-file VERSION/reset commit to main, then tag runs after that SHA's own full ci-post is green and tags that exact bump commit via dispatch_sha, builds and attaches the tarballs and catalog assets, and notifies the hub. The release gate now takes the phase, dispatch SHA and a dry-run flag, writes a receipt that is uploaded as an artifact, and the version plan comes from ci_release_gate.py plan instead of release_version.py next. The tag phase never writes VERSION or creates a commit, and retries reuse an existing tag only if it sits at that same SHA, so concurrent tag runs racing to create the same GitHub Release is the part most likely to fail.
Worth a look
- Concurrent tag runs can fail while creating the same release —
.github/workflows/release.yml:378· Escalate · medium- unverified hypothesis — survived the adversarial refute pass; needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 3821 functions depend on the 3563 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
ci_workflow_findings()— 93 callers, 25 callees - worse:
admit_rule_findings()— 9 callers, 9 callees - worse:
_job_block()— 22 callers, 3 callees - worse:
build_diff()— 8 callers, 8 callees - worse:
do_check()— 3 callers, 17 callees - worse:
native_run_spec_digest()— 17 callers, 3 callees - new:
parse()— 6 callers, 8 callees - worse:
_holds_credentials()— 6 callers, 7 callees - …and 28 more — each is listed as a finding
Verification — 3821 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 3668 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
1414 of 1414 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety (
noncode-change,uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.
benchmarks/tests/test_aggregate.py— full-run-safetybenchmarks/tests/test_check_provenance.py— full-run-safetybenchmarks/tests/test_compare.py— full-run-safetybenchmarks/tests/test_public_names.py— full-run-safetybenchmarks/tests/test_run_all.py— full-run-safetybenchmarks/tests/test_run_strategy_feeds.py— full-run-safetydocker/pineforge_sink_finalize_test.py— full-run-safetydocker/pineforge_sink_replay_rq_test.py— full-run-safetydocker/pineforge_sink_replay_test.py— full-run-safetydocker/pineforge_sink_to_json_test.py— full-run-safetydocker/quota_columns_test.py— full-run-safetydocker/quota_convert_test.py— full-run-safetydocker/quota_finalize_test.py— full-run-safetydocker/replay_rq_engine_test.py— full-run-safetydocker/run_json_artifact_window_test.py— full-run-safetydocker/run_json_bin_test.py— full-run-safetydocker/run_json_bounded_transport_test.py— full-run-safetydocker/run_json_bulk_equity_fuzz_test.py— full-run-safetydocker/run_json_bulk_equity_test.py— full-run-safetydocker/run_json_codes_test.py— full-run-safetydocker/run_json_converter_metadata_test.py— full-run-safetydocker/run_json_converter_session_test.py— full-run-safetydocker/run_json_diagnostics_test.py— full-run-safetydocker/run_json_equity_scan_test.py— full-run-safetydocker/run_json_fused_test.py— full-run-safetydocker/run_json_harnesses_test.py— full-run-safetydocker/run_json_metrics_only_test.py— full-run-safetydocker/run_json_misc2_test.py— full-run-safetydocker/run_json_misc_test.py— full-run-safetydocker/run_json_pfbar_test.py— full-run-safetydocker/run_json_provenance_test.py— full-run-safetydocker/run_json_quota_supervisor_test.py— full-run-safetydocker/run_json_replay_test.py— full-run-safetydocker/run_json_selected_sink_test.py— full-run-safetydocker/run_json_sink_lexical_test.py— full-run-safetydocker/run_json_sink_summary_test.py— full-run-safetydocker/run_json_sink_test.py— full-run-safetydocker/run_json_sink_vectors_test.py— full-run-safetydocker/run_json_spool_work_test.py— full-run-safetydocker/run_json_stringio_test.py— full-run-safetydocker/run_json_symbol_feeds_test.py— full-run-safetydocker/run_json_syminfo_test.py— full-run-safetydocker/run_json_trade_scan_test.py— full-run-safetydocker/run_json_window_budget_test.py— full-run-safetydocker/run_json_wrapper_work_test.py— full-run-safetyinclude/pineforge/native_run_spec.hpp— full-run-safetyscripts/derive_corpus_feeds_self_test.py— full-run-safetyscripts/fingerprint_self_test.py— full-run-safetyscripts/run_corpus_self_test.py— full-run-safetyscripts/test_abi_receipt_skips.py— full-run-safety- … and 1364 more
non-code file(s) changed (
.github/workflows/release.yml,CHANGELOG.md,CMakeLists.txt,ci/cr150-source-bounds.yaml,ci/cr150-source-sites.json…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
.github/workflows/release.yml,CHANGELOG.md,docs/adr/0001-kernel-adapter-boundary.md,docs/ci.md,docs/design/native-feature-parity.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
AGENTS.md§ What this repo is (lines 5-11): references changed symbolssourceAGENTS.md§ Build (lines 12-25): references changed symbolsreceiptAGENTS.md§ Layout (lines 26-41): references changed symbolssourceAGENTS.md§ What CI runs, per trigger (lines 46-70): references changed symbolsdiagnosticAGENTS.md§ Local-only tests (no CI job runs them) (lines 71-82): references changed symbolsselectedAGENTS.md§ Conventions (lines 103-116): references changed symbolssourceAGENTS.md§ Safety rules (lines 117-127): references changed symbolssourceCLAUDE.md§ What this repo is (lines 5-11): references changed symbolssourceCLAUDE.md§ Build (lines 12-25): references changed symbolsreceiptCLAUDE.md§ Layout (lines 26-41): references changed symbolssource
…and 10 more.
· 20 grounded finding(s) anchored inline below; 11 more finding(s) on lines outside this diff (see the check run); 5 additional anchorable finding(s) not shown (cap).
|
|
||
| def _release_split_toolchain_reader_is_pinned(workflow: str, body: str) -> bool: | ||
| """Admit only the reviewed OIDC reader; scan all other raw credential inputs.""" | ||
| def _release_split_current_toolchain_reader_is_pinned(workflow: str, body: str) -> bool: |
There was a problem hiding this comment.
_release_split_current_toolchain_reader_is_pinned()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return not _holds_credentials(workflow, token_only_body, strict=True) | ||
|
|
||
|
|
||
| def _release_split_contract_findings(workflow: str, mode: str) -> list[str]: |
There was a problem hiding this comment.
_release_split_contract_findings()
fans out to 11 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| @@ -5084,17 +5534,71 @@ def release_rulesets_probe_workflow_findings(workflow: str, release: str) -> lis | |||
|
|
|||
|
|
|||
| def release_bump_workflow_findings(workflow: str) -> list[str]: | |||
There was a problem hiding this comment.
release_bump_workflow_findings()
fans out to 10 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| _fields(run, ("created_at", "updated_at", "run_started_at", "html_url"))) | ||
|
|
||
|
|
||
| def _observe_run(api: API, repo: str, run: dict, record: dict) -> None: |
There was a problem hiding this comment.
_observe_run()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| validation["outcome"] = "pending" | ||
|
|
||
|
|
||
| def _revalidate_runs(api: _GateReads) -> None: |
There was a problem hiding this comment.
_revalidate_runs()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| FOUR_HOURS_MS = 4 * HOUR_MS | ||
|
|
||
|
|
||
| def grid_neighbours(start_ms, offset_hours): |
There was a problem hiding this comment.
grid_neighbours()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| std::vector<std::uint64_t> lengths; | ||
| }; | ||
|
|
||
| Parsed parse(Memory& memory, const std::string& digest, std::uint64_t cap, |
There was a problem hiding this comment.
parse()
fans out to 8 callees (efferent coupling); 6 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| } // namespace | ||
|
|
||
| int main() { |
There was a problem hiding this comment.
main()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| self.assertEqual(manifest["helper"]["bytes"], str(len(data))) | ||
| self.assertEqual(manifest["helper"]["sha256"], sha(data)) | ||
|
|
||
| def check_install(prefix, expected_sha, dirty): |
There was a problem hiding this comment.
check_install()
high coupling complexity (Ca·Ce = 25).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| CHECK(!result.plan_attempted); | ||
| } | ||
|
|
||
| void equality_includes_headers_all_roles_and_actual_newlines() { |
There was a problem hiding this comment.
equality_includes_headers_all_roles_and_actual_newlines()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
Superseded by #822, which carries the 1.11.1 batch on a new train head. |
One combined change for v1.11.1, gated once instead of once per pull request: plain merges of three reviewed pull requests onto main
f5e929df, with no edits of their own.Members, each at the head that was reviewed:
6955f22fb114a1e5c244973a5a7ad1cf357a8763: places the closed-trade read cache by row count, which removes the slowdown on large closed-trade histories.33125563af08d8db33bc00f8b892e7f27074ef1c: adds the selected-capacity admission helper and its descriptor specification.5060c18f221c50308bd28fc331d555d92766025f: separates the release bump from tag admission and keeps the evidence each decision rests on.The combined head is tested as one unit: the full hosted CI, one full verification, one parity sweep, and a review of the combined diff that checks how the members interact. The original pull requests are closed with a link to this one when it merges.
🤖 Generated with Claude Code