Skip to content

v1.11.1 engine train: #797, #801, #789 - #808

Closed
luisleo526 wants to merge 29 commits into
mainfrom
release-train-1.11.1-b
Closed

luisleo526 wants to merge 29 commits into
mainfrom
release-train-1.11.1-b

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

One combined change for v1.11.1, gated once instead of once per pull request: plain merges of three reviewed pull requests onto main f5e929df, with no edits of their own.

Members, each at the head that was reviewed:

The combined head is tested as one unit: the full hosted CI, one full verification, one parity sweep, and a review of the combined diff that checks how the members interact. The original pull requests are closed with a link to this one when it merges.

🤖 Generated with Claude Code

luisleo526 and others added 29 commits November 7, 2026 21:29
Exercise the version-only reset before tagging, preserving tagged behavior.
Add refusal controls for parent count, extra paths, version direction,
catalog bytes, existing tags and inherited release stamp constraints.
Require each release phase to use its own exact-commit CI evidence. Keep the
bump untagged until its checks finish, then authenticate and tag that same
commit. Validate the two-file transition and preserve catalog assets.
Record advisory verification and complete workflow attempt history, bind the
decision to received response bytes, and refuse changed run metadata before
admission. Keep release receipts on refusal and honor declared catalog paths.
Use timezone-aware UTC and keep exact output assertions aligned with each
test input. Preserve every release safety condition and mutation control.
Cover authenticated advisory creators and first-attempt actor refusals.
Reject malformed advisory states before logging them, retain sanitized
unexpected-exception receipts, and distinguish invalid triggering actors
from changed attempt metadata. Keep parity as the only required status.
Carry both strict admission forms in the hosted release workflow, retaining
caller identity, artifact custody and exact-commit release evidence.
Keep historical matrix inputs independent from the phased release workflow
and require both phase and dry-run boundaries on downstream publishing jobs.
Choose the expected guard and message from the complete dispatch declaration,
while retaining strict workflow acceptance and both dry-run boundaries.
…hosted jobs

WorkflowPinTests reads the #768 matrix rebuild of release.yml for its
hosted-matrix contracts. That rebuild keeps the shipped two-phase release
job, a combination the validator correctly refuses, and its single
prebuilt job has no tag-phase guard. Four checks therefore failed. They
now read the shipped workflow:

- test_two_release_phases_stop_before_publication_until_tag_phase pins
  the exact job-level tag-phase if: of prebuilt-x86, prebuilt-arm and
  upload, and the closed job list. The notify-hub, gate, step-condition
  and writer pins are unchanged.
- test_prebuilt_requires_release_success_without_failure_bypasses holds
  the release-success pin for the release job, both build jobs and the
  publishing upload job: needs: release on each build job, upload's exact
  needs: [release, prebuilt-x86, prebuilt-arm], no always(), !cancelled()
  or continue-on-error: outside comments, and no
  release_bump_workflow_findings.
- test_prebuilt_failure_bypass_mutations_are_refused, both subtests,
  inserts each bypass into each build job and into upload and requires
  the same refusal. Each of those jobs first passes with the shipped
  "no continue-on-error" comment in it, so the pin is still shown to
  read keys, not words.

The validator compares every split job header, so the pin now scans the
build jobs and upload before the release job, and a refusal names its
job and bypass. The other WorkflowPinTests tests keep reading the matrix
rebuild.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scripts that read their closed trades again and again (strategy.closedtrades accessors in a loop) were about 1.9 to 2.0 times slower than on 1.10.0 on histories of 15,500 and 39,456 closed trades. Two things made them slow: every cache hit rewrote the rank of the other entries of its set, and four-way sets spread consecutive trades over four times the pages their rows need.

The cache now follows the number of closed trades. While the ledger holds at most as many rows as the cache has entries (215,070 by default), row n owns slot n: one way, one probe per lookup, and only the pages the rows reach are committed (25,000 closed trades hold 98 pages, 7.8 MB, where four-way sets held 391 pages, 31.2 MB). With more rows the four-way sets apply as before. An append that takes the ledger past the cache's rows, or a rewind back to them, drops every cached row and keeps the allocated pages. In a set, a hit on the most recent entry changes nothing and a hit on any other entry ages only the entries more recent than it, so a full set replaces the entry touched longest ago, as the documentation says. A miss is probed and counted once. Every value a script reads is unchanged: the reports of a 15,500-trade script are identical field by field.

Measured on an aarch64 build host, one pinned core, engine time against 1.10.0 built with the same compiler: a script that scans 15,500 closed trades again and again takes 1.03 of the 1.10.0 time (five rounds; main before this change 1.92); a script that only opens and closes trades is unchanged by this change (1.10 on 150,000 bars, as on main). With 250,000 closed trades, above the cache's rows, a script that scans all of them every 6,000 bars takes 40 s, against 27 s on main before this change and 138 s on 1.10.0 (two rounds). Timings on x86-64 were not measured.

Tests: the placement oracle in tests/ledger_default_cache_rows.hpp now takes the number of rows the ledger holds, and nine tests that compared resident bytes or entry counts with it state both placements; three new tests pin the least-recently-used rule, the single probe of a miss and the placement change.

Documentation: docs/pages/memory-management.md and the header comment describe the placement by row count. Two sentences of the memory page leave in the ledger read cache row: the sentence that began "Storage is private: a global ordinal chooses one four-way set" (introduced by ba14721) is rewritten to state the placement for both populations, and "Page growth preserves entries through the 64/128/256 boundaries." (introduced by 69cddc9) now says that growth preserves entries through every page boundary, every 256 rows within the cache's rows.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Preserve phase-aware release admission while incorporating the current
main toolchain contract. Keep both Eigen flag constants and the mode
parameter, and update the planned-phase fixture to the typed PATH flag.

Co-Authored-By: Codex <[email protected]>
Bring published PR #797 onto engine main a447232 without rewriting
the published kSA commit 653be41. Retain the release and CI changes
and the existing closed-trade cache implementation and regression inputs.

Exact-head CI, parity and official review remain required on this merge.
Retain the tagged VERSION and catalog baseline before exact-head validation.

Co-Authored-By: Codex <[email protected]>
Complete the selected-capacity production entry and add
pineforge-selected-admit, a small command-line program that checks a caller's
capacity bounds on the Source and the selected inputs of a run before the
first writer consumes them.

Production entry: the five selected-capacity observers are bound through the
native materializer, and actual Source records and CSV bytes are accounted
with the existing reserve primitive. Input-sensitive tests cover the field
limits, the exact failure arguments, the byte accounting and the production
binding.

Helper: it reads a versioned request descriptor (tools/selected_admit/
DESCRIPTOR-V1.md specifies the request, the result and the manifest), takes
custody of each input by descriptor number, size and digest, parses it with
the existing selected-capacity reader, calls the product entry, and answers
with one result document and an exit status. The Source stage forwards each
record only after it is admitted, and refuses a forward descriptor that is the
same file as its stdout or stderr (exit 41); the Selected stage counts the
selected pass and retains nothing. The helper and its manifest (size, sha256,
build identity, schema digests, stages) install to bin/ and share/pineforge/.
The sha256 of the spec is compiled into the manifest, so a change to the spec
changes the manifest. The spec also lists what the Source stage holds in
memory (the record window and the reader's copy of one field, each up to the
record cap, a chunk and the process base) with one measured figure, and what
the helper reads from the environment and the file system: GLIBCXX_TUNABLES at
start, and, for a calendar zone that is not UTC-like, TZ and, per libc, TZDIR
and the zone file under the zone database root.

Tests: descriptor, custody, stage and manifest tests, the Source-stage parity
test, an import-list test and a calendar test. The import-list test reads the
undefined symbols of the built helper with nm and fails if it imports one of
its list of 41 process, thread, session, socket, descriptor-duplication,
signalling, system-call or library-loading functions, other than dlopen (one
RTLD_NOLOAD lookup of the engine's own image) and recvmsg (the run-quota
endpoint). The two allowed names are pinned in the test, a glibc fortified or
internal spelling of a forbidden name counts as the name, an nm line that
cannot be parsed stops it, and a listing without pread, poll and
__cxa_begin_catch, which the helper builds measured so far import, is rejected
as empty, truncated or unrelated; having them proves neither completeness nor
which file nm read. The calendar test runs the Selected stage with New York
four-hour script bars across the two clock changes of 2024: the open of the
date's own season is admitted, the open of the other season is refused with
the neighbouring bar opens that civil-time arithmetic gives, and a zone
database root that holds no zone makes the zone unsupported.

Inventory: the allocation-site inventory gets the three new Source-stage
groups and five site rows, the unreviewed ceiling goes from 1410 to 1413 and
the exact-union pin to 1356 + 57 + 184, the five selected limit routes of the
ownership manifest are corrected, and the CMakeLists.txt line citations in
five documents are re-anchored (three citations that named a neighbouring line
now name the line they describe).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The capture counter starts after the ledger cache has been populated, so
resident cache-page growth must not be required in its working-heap peak.
Restore the symmetric fixed-window comparison while retaining the exact
row-count-dependent cache residency assertions and all output oracles.
Retain the row-count-derived page-growth upper envelope, since Release
can allocate cache pages during capture. Do not require that growth in
the measured peak when pages are already resident. Keep the exact cache
residency, scratch-window, output and profile checks unchanged.
…utral names

The registration file of the helper's unit tests, the fence around its script
tests in tests/CMakeLists.txt, and the CTest label of all seven tests carried a
name derived from a work card. They are now selected-admit-helper and
selected_admit_helper. The registrations are otherwise unchanged, and nothing
else in the tree named the old file or label.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The private-fixture reads check reads the add_test arguments of tests/CMakeLists.txt
and refuses a variable that starts with _pf_ unless it saw that variable set in
the files it reads. The manifest test took its expected stages from
_pf_sa_stages_json, which the root CMakeLists.txt sets, so the check stopped on it
and test_private_fixture_reads and its controls failed in every build that runs
the full suite. The pull request always carries tools/selected_admit/stage_source.cpp,
so the root sets that variable to ["source","selected"]; the test now gets
exactly that text, and it still compares it with the stages in the built
helper's manifest.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…rule

CMakeLists.txt takes the sha256 of tools/selected_admit/DESCRIPTOR-V1.md into the
selected-admit helper's manifest at configure time, so a change to that page changes
a build output. The docs-only rule called every Markdown file outside its excluded
trees documentation, and test_no_cmake_file_reads_documentation (the docs-only-tests
stage of preflight) failed on exactly this page: Lists differ:
['tools/selected_admit/DESCRIPTOR-V1.md'] != []. The page joins PROOF_READS, the
list of documentation inputs that proof jobs read, so a change to it runs every
proof leg. The pinned list in the rule's test, the scan test that now requires the
page among the CMake reads it must see, and the sentence in docs/ci.md that counts
the excluded files are updated with it. A new test replays the page in real git
histories: an edit, a rename away, a rename onto its path and a deletion all run the
proof legs, and another Markdown page of the same directory stays documentation.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ision evidence

docs/ci.md: replaces the release-bump exemption sentence introduced by 72d9fd5 (#486), 'When there is no qualifying merged PR for the current main sha, phase 1 permits one release bump only when all three conditions hold:', with the phased release-bump and tag-admission wording of #789.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Comment on lines +9295 to +9296
'release.yml job prebuilt-x86 must not receive the protected release '
'environment or release credentials',
Comment on lines +9299 to +9300
'release.yml must keep the exact two-phase release writer, '
'untagged bump push and same-SHA tag retry',
Comment on lines +9301 to +9302
'release.yml must keep the unconditional release gate with its release '
'App token for the App-only main pull_request barrier',
Comment on lines +9303 to +9304
'release.yml must always retain the exact release gate receipt artifact '
'before planning or publishing',
Comment on lines +9332 to +9333
'release.yml job prebuilt-x86 must not receive the protected release '
'environment or release credentials',
Comment on lines +9335 to +9336
'release.yml must keep the exact two-phase release writer, '
'untagged bump push and same-SHA tag retry',
Comment on lines +9337 to +9338
'release.yml must keep the unconditional release gate with its release '
'App token for the App-only main pull_request barrier',
Comment on lines +9339 to +9340
'release.yml must always retain the exact release gate receipt artifact '
'before planning or publishing',
Comment on lines +9818 to +9819
'release.yml must keep the exact two-phase release writer, '
'untagged bump push and same-SHA tag retry',
Comment on lines +9820 to +9821
'release.yml must keep the unconditional release gate '
'with its release App token for the App-only main pull_request barrier',

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; no formal proof of the changed code.


Graphify review — findings

Splits the release workflow into two dispatch phases: bump gates the dispatch SHA and pushes only the untagged two-file VERSION/reset commit to main, then tag runs after that SHA's own full ci-post is green and tags that exact bump commit via dispatch_sha, builds and attaches the tarballs and catalog assets, and notifies the hub. The release gate now takes the phase, dispatch SHA and a dry-run flag, writes a receipt that is uploaded as an artifact, and the version plan comes from ci_release_gate.py plan instead of release_version.py next. The tag phase never writes VERSION or creates a commit, and retries reuse an existing tag only if it sits at that same SHA, so concurrent tag runs racing to create the same GitHub Release is the part most likely to fail.

Worth a look

  • Concurrent tag runs can fail while creating the same release — .github/workflows/release.yml:378 · Escalate · medium
    • unverified hypothesis — survived the adversarial refute pass; needs human review

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 3821 functions depend on the 3563 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: ci_workflow_findings() — 93 callers, 25 callees
  • worse: admit_rule_findings() — 9 callers, 9 callees
  • worse: _job_block() — 22 callers, 3 callees
  • worse: build_diff() — 8 callers, 8 callees
  • worse: do_check() — 3 callers, 17 callees
  • worse: native_run_spec_digest() — 17 callers, 3 callees
  • new: parse() — 6 callers, 8 callees
  • worse: _holds_credentials() — 6 callers, 7 callees
  • …and 28 more — each is listed as a finding

Verification — 3821 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 3668 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

1414 of 1414 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • benchmarks/tests/test_aggregate.py — full-run-safety
  • benchmarks/tests/test_check_provenance.py — full-run-safety
  • benchmarks/tests/test_compare.py — full-run-safety
  • benchmarks/tests/test_public_names.py — full-run-safety
  • benchmarks/tests/test_run_all.py — full-run-safety
  • benchmarks/tests/test_run_strategy_feeds.py — full-run-safety
  • docker/pineforge_sink_finalize_test.py — full-run-safety
  • docker/pineforge_sink_replay_rq_test.py — full-run-safety
  • docker/pineforge_sink_replay_test.py — full-run-safety
  • docker/pineforge_sink_to_json_test.py — full-run-safety
  • docker/quota_columns_test.py — full-run-safety
  • docker/quota_convert_test.py — full-run-safety
  • docker/quota_finalize_test.py — full-run-safety
  • docker/replay_rq_engine_test.py — full-run-safety
  • docker/run_json_artifact_window_test.py — full-run-safety
  • docker/run_json_bin_test.py — full-run-safety
  • docker/run_json_bounded_transport_test.py — full-run-safety
  • docker/run_json_bulk_equity_fuzz_test.py — full-run-safety
  • docker/run_json_bulk_equity_test.py — full-run-safety
  • docker/run_json_codes_test.py — full-run-safety
  • docker/run_json_converter_metadata_test.py — full-run-safety
  • docker/run_json_converter_session_test.py — full-run-safety
  • docker/run_json_diagnostics_test.py — full-run-safety
  • docker/run_json_equity_scan_test.py — full-run-safety
  • docker/run_json_fused_test.py — full-run-safety
  • docker/run_json_harnesses_test.py — full-run-safety
  • docker/run_json_metrics_only_test.py — full-run-safety
  • docker/run_json_misc2_test.py — full-run-safety
  • docker/run_json_misc_test.py — full-run-safety
  • docker/run_json_pfbar_test.py — full-run-safety
  • docker/run_json_provenance_test.py — full-run-safety
  • docker/run_json_quota_supervisor_test.py — full-run-safety
  • docker/run_json_replay_test.py — full-run-safety
  • docker/run_json_selected_sink_test.py — full-run-safety
  • docker/run_json_sink_lexical_test.py — full-run-safety
  • docker/run_json_sink_summary_test.py — full-run-safety
  • docker/run_json_sink_test.py — full-run-safety
  • docker/run_json_sink_vectors_test.py — full-run-safety
  • docker/run_json_spool_work_test.py — full-run-safety
  • docker/run_json_stringio_test.py — full-run-safety
  • docker/run_json_symbol_feeds_test.py — full-run-safety
  • docker/run_json_syminfo_test.py — full-run-safety
  • docker/run_json_trade_scan_test.py — full-run-safety
  • docker/run_json_window_budget_test.py — full-run-safety
  • docker/run_json_wrapper_work_test.py — full-run-safety
  • include/pineforge/native_run_spec.hpp — full-run-safety
  • scripts/derive_corpus_feeds_self_test.py — full-run-safety
  • scripts/fingerprint_self_test.py — full-run-safety
  • scripts/run_corpus_self_test.py — full-run-safety
  • scripts/test_abi_receipt_skips.py — full-run-safety
  • … and 1364 more

non-code file(s) changed (.github/workflows/release.yml, CHANGELOG.md, CMakeLists.txt, ci/cr150-source-bounds.yaml, ci/cr150-source-sites.json …) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (.github/workflows/release.yml, CHANGELOG.md, docs/adr/0001-kernel-adapter-boundary.md, docs/ci.md, docs/design/native-feature-parity.md …) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

…and 10 more.

· 20 grounded finding(s) anchored inline below; 11 more finding(s) on lines outside this diff (see the check run); 5 additional anchorable finding(s) not shown (cap).

Comment thread scripts/ci_preflight.py

def _release_split_toolchain_reader_is_pinned(workflow: str, body: str) -> bool:
"""Admit only the reviewed OIDC reader; scan all other raw credential inputs."""
def _release_split_current_toolchain_reader_is_pinned(workflow: str, body: str) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _release_split_current_toolchain_reader_is_pinned()

fans out to 9 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread scripts/ci_preflight.py
return not _holds_credentials(workflow, token_only_body, strict=True)


def _release_split_contract_findings(workflow: str, mode: str) -> list[str]:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _release_split_contract_findings()

fans out to 11 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Comment thread scripts/ci_preflight.py
@@ -5084,17 +5534,71 @@ def release_rulesets_probe_workflow_findings(workflow: str, release: str) -> lis


def release_bump_workflow_findings(workflow: str) -> list[str]:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — release_bump_workflow_findings()

fans out to 10 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

_fields(run, ("created_at", "updated_at", "run_started_at", "html_url")))


def _observe_run(api: API, repo: str, run: dict, record: dict) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _observe_run()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

validation["outcome"] = "pending"


def _revalidate_runs(api: _GateReads) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — _revalidate_runs()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

FOUR_HOURS_MS = 4 * HOUR_MS


def grid_neighbours(start_ms, offset_hours):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — grid_neighbours()

high coupling complexity (Ca·Ce = 12).

Grounded coupling-delta finding (deterministic), not an LLM guess.

std::vector<std::uint64_t> lengths;
};

Parsed parse(Memory& memory, const std::string& digest, std::uint64_t cap,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — parse()

fans out to 8 callees (efferent coupling); 6 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

}
} // namespace

int main() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — main()

fans out to 9 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

self.assertEqual(manifest["helper"]["bytes"], str(len(data)))
self.assertEqual(manifest["helper"]["sha256"], sha(data))

def check_install(prefix, expected_sha, dirty):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — check_install()

high coupling complexity (Ca·Ce = 25).

Grounded coupling-delta finding (deterministic), not an LLM guess.

CHECK(!result.plan_attempted);
}

void equality_includes_headers_all_roles_and_actual_newlines() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — equality_includes_headers_all_roles_and_actual_newlines()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

@luisleo526

Copy link
Copy Markdown
Collaborator Author

Superseded by #822, which carries the 1.11.1 batch on a new train head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant