Candidate: add the selected-capacity admission helper - #801

Candidate build, not ready to merge. This pull request is four commits made on main as it stood at a4472324. Its head (33125563) is a member of the 1.11.1 engine train: the newer main (f5e929df) is merged on the train branch, not in this pull request, which has no merge commit. The results below ran on these four commits. The four commits are: the helper with its tests (9dab4ef8); a second commit that only gives the helper's CMake registration file, its fence and its test label neutral names (ecff2566); a third that changes one line, the expected stages the manifest test is given (e7105555); and a fourth that lists the helper's spec page among the pages a proof job reads, in the rule that decides which changes skip the proof jobs (33125563). It completes the selected-capacity production entry that main only declares today and adds pineforge-selected-admit, a small command-line program that checks a caller's capacity bounds on the Source and the selected inputs of a run before the first writer consumes them. It merges only through the normal gates: CI green, the parity check, and a review of the final head.

What it adds

• Production entry. The five selected-capacity observers are bound through the native materializer, and actual Source records and CSV bytes are accounted with the existing reserve primitive. Input-sensitive tests cover the field limits, the exact failure arguments, the byte accounting and the production binding.
• Helper. pineforge-selected-admit reads a versioned request descriptor and its inputs from inherited file descriptors, takes custody of each input by descriptor number, size and digest, applies the caller's five capacity bounds with the product entry, and prints one result document with an exit status per outcome. The Source stage forwards each record only after it is admitted, and refuses a forward descriptor that is the same file as its stdout or stderr (exit 41, before any read); the Selected stage counts the selected pass and retains nothing. tools/selected_admit/DESCRIPTOR-V1.md specifies the request, the result and the manifest; its two example result documents show what the helper prints. The spec also lists what the Source stage holds in memory (the record window and the reader's copy of one field, each up to the record cap, a chunk and the process base) with one measured figure, and what the helper reads from the environment and the file system: GLIBCXX_TUNABLES at start and, for a calendar zone that is not UTC-like, TZ and, per C library, TZDIR and the zone file under the zone database directory.
• Install. The helper installs to bin/pineforge-selected-admit and its manifest to share/pineforge/pineforge-selected-admit.manifest.json. The manifest records the size and sha256 of the helper, the build identity and the sha256 of the spec (descriptor.spec_sha256), which is taken at configure time, and the stages the helper offers.
• Manifest test. test_selected_admit_manifest configures, builds and installs the engine in a second tree (always a Release build), then checks the installed manifest against the installed helper: its --describe output, the size and sha256 of the file, the build identity, the spec and failure-catalog digests, the stages, a probe of the Source stage and five Selected-stage requests: three carry an identity the helper must refuse (exit 40), two carry the matching identity and are answered no_requested_fields (exit 10). The requests use empty bounds and a header-only CSV; the test does not exercise admission with real bounds and records.
• Import-list test. test_selected_admit_imports reads the undefined symbols of the test-identity build of the helper (the same sources with a fixed test identity unit; this test does not read the shipped target) with nm and fails if it imports one of its list of 41 process, thread, session, socket, descriptor-duplication, signalling, system-call or library-loading functions, other than dlopen (one RTLD_NOLOAD lookup of the engine's own image) and recvmsg (the run-quota endpoint); a glibc fortified (__name_chk) or internal (__name) spelling of a name counts as the name. The two allowed names are pinned in the test, a line of the nm listing that cannot be parsed stops it and the second nm option is not tried, and a listing without pread, poll and __cxa_begin_catch (a fortified spelling counts), which the helper builds measured so far import, is rejected as empty, truncated or unrelated; having them proves neither that the listing is complete nor which file nm read. It reads symbol tables only.
• Calendar test. test_selected_admit_calendar runs the Selected stage with a New York calendar of four-hour script bars, which are anchored at the session open, across the two clock changes of 2024. The open of the date's own season is admitted; the open of the other season is refused with boundary_unaligned and the neighbouring bar opens that civil-time arithmetic gives; a zone database directory that holds no zone makes the zone unsupported; a UTC calendar ignores the directory. Every expected instant is derived in the test. It needs the system zone database and fails, not skips, without it. Only a glibc build of the engine reads TZDIR, so on any other host the two missing-directory rows expect the admitted answer.
• Inventory. The allocation-site inventory gets the three new Source-stage groups and five site rows. The unreviewed ceiling goes from 1410 to 1413 and the exact-union pin to 1356 + 57 + 184. The five selected limit routes of the ownership manifest are corrected, and the CMakeLists.txt line citations in five documents are re-anchored (three citations that named a neighbouring line now name the line they describe).
• Docs-only rule. CMakeLists.txt takes the sha256 of the spec page tools/selected_admit/DESCRIPTOR-V1.md into the helper's manifest at configure time, so a change to that page changes a build output. The page is now listed in PROOF_READS in scripts/ci_docs_only.py (the pages a proof job reads, which never count as documentation), its pinned list and the scan that finds pages read by CMake are updated, and docs/ci.md counts six such files instead of five.

Checks run before opening

Each result names the head it ran on and says what kind it is. Historical results ran on the first commit and are not repeated on the later heads. Focused results are a chosen set run on the named head. CLOUD results are full verify profiles on x86_64 cloud machines. A historical result is not evidence for a later head unless a focused or CLOUD item below says so.

Historical: first commit (9dab4ef8), one Linux arm64 host

• These ran on the first commit, except the controls that must fail, which ran on earlier builds with the same test and gate sources (the only changes to those sources since are docstrings and comments of the import-list test and wording of the spec). The two calendar controls that use the real helper, and the list of 242 imported names below, come from an earlier build of the helper, not from this commit's build.
• In Release, Debug and ASan+UBSan builds, the seven helper tests and the six selected-capacity tests pass in each build; the sanitizer flags are in the helper's compile commands in the third.
• The new stages test runs the helper with its stdout, then its stderr, as the forward pipe. Without the refusal it fails (with stdout as the forward pipe the forwarded rows come before the result line; with stderr the helper exits 0); with it both runs exit 41 with the input unread, and a separate forward pipe still forwards every record. Dropping the comparison with either stream from the refusal fails the test for that stream.
• The import-list test passes against the test-identity build of the helper, against the 242 names an earlier build of the helper imports, and against a program that imports only the two allowed names and the three sentinel names. It fails, with its own message, against programs that import fork; dup2, pthread_create, socket and system; syscall; a fortified recvfrom (__recvfrom_chk, which the version without the spelling rule passed); or only puts; against a stand-in nm that prints a line that is not an nm line; and against a copy of the test whose allow-list has a third name.
• The calendar test fails, with assertion failures, against four helpers that must not pass it: one that always admits, one that always refuses, the real helper (an earlier build) with a zone database whose New York file is a copy of the UTC file, and the real helper (an earlier build) with TZDIR forced to a missing directory.
• The allocation-site gate reports 1597 entries and no finding, and its 91 tests pass. On an earlier build of the same gate files, a copy of the gate's bounds file under ci/ with a ceiling of 1412 failed with exactly one ceiling finding, and the previous exact-union pin failed with 1597 != 1594 and nothing else.
• The documentation and comment guards of the preflight pass.

Focused: second commit (ecff2566), one Linux arm64 host

• The second commit changes no source and no test, only those names. The same 26 documentation and source guards and the net documentation-revert check ran and ended as on the first commit (the native-versions check skips for want of private fixtures, and the allocation-site gate is an open-only gate, not a proof of flat memory); the registration checker for the per-area CMake files and its controls pass; the eight preflight slices and their cover row pass; ctest -N -L selected_admit_helper lists the seven helper rows (registration only: nothing was built or run for them) and the old label lists none; the list of all tests with their commands and properties is the first commit's apart from that label. The registration file now sorts earlier, so 73 rows get new CTest numbers; no name, command or property changes.

CLOUD: second commit (ecff2566), x86_64 cloud machines (spot instances, 112 vCPU)

• The debug, kernel, release and sanitizer (ASan+UBSan) builds each failed on the same two rows of the full suite and no others: test_private_fixture_reads and test_private_fixture_reads_controls (2 of 1469 rows in debug, 2 of 784 in kernel, 2 of 1581 in release, 2 of 1440 in the sanitizer build). The private-fixture reads check refuses a CMake variable that begins with _pf_ in a test command unless it saw that variable set in the files it reads, and the manifest test was registered with --expected-stages "${_pf_sa_stages_json}". That registration comes from the first commit and the second leaves it as it was; none of the focused runs above included these two rows. In the release and sanitizer builds the seven helper tests, the registration checker and its controls passed.
• The preflight failed at its docs-only stage: test_no_cmake_file_reads_documentation in scripts/test_ci_docs_only.py reported Lists differ: ['tools/selected_admit/DESCRIPTOR-V1.md'] != [] (the fourth commit fixes this). The unittest shards of the same preflight (547 tests in 8 shards) passed, and so did the documentation build.
• The corpus parity step passed (all 312 corpus probes match the recorded baseline: 311 excellent and one classed as an anomaly) and so did the subset step (54 of 312 probes).

Focused: third commit (e7105555), one Linux arm64 host, Release

• The third commit replaces ${_pf_sa_stages_json} in that registration by the literal ["source","selected"]. The literal mirrors the value that the CMakeLists.txt in the repository's base directory sets for the helper's stages: ["selected"] at line 348, replaced at line 352 by ["source","selected"] when tools/selected_admit/stage_source.cpp exists, as it does in this pull request. The test still compares the literal with the stages of the manifest of the built helper, so a change on either side fails it.
• test_private_fixture_reads and test_private_fixture_reads_controls, the two rows that failed in the CLOUD builds of the second commit, pass; the configured manifest test command carries exactly ["source","selected"].
• The manifest test passes. Given ["selected"] or ["selected","source"] for this tree it fails at the comparison with the manifest's stages; with the stage list changed in a copy of that CMakeLists.txt it fails at the comparison of the manifest with the helper's own --describe output; on a copy of the tree without tools/selected_admit/stage_source.cpp it fails given ["source","selected"] and passes given ["selected"].
• The registration checker for the per-area CMake files and its controls pass; the eight preflight slices and their cover row pass; ctest -N -L selected_admit_helper lists the same seven rows and the old label lists none; the 26 documentation and source guards and the net documentation-revert check end as on the second commit (the native-versions check skips for want of private fixtures).

CLOUD: third commit (e7105555), x86_64 cloud machines

• The debug, kernel, corpus parity and subset steps passed: 1469 rows in debug and 784 in kernel, all passed (the two private-fixture rows included); the corpus parity step matched all 312 probes (311 excellent, one classed as an anomaly) and the 54-probe subset matched.
• A parity sweep over the whole pinned population on this commit's tree measured 134 of 134 cases (7989 graded) with no coverage lost, no regression and no leaver against the baseline. When this was written the result had not been posted as a status.
• The documentation step of this commit passed; its preflight, release and sanitizer steps were still running when this was written, and the preflight is expected to fail at the docs-only stage until the fourth commit.

Focused: fourth commit (33125563), one Linux arm64 host

• The fourth commit lists tools/selected_admit/DESCRIPTOR-V1.md among the pages a proof job reads (PROOF_READS in scripts/ci_docs_only.py), with the check that reads it: CMakeLists.txt takes its sha256 into the helper's manifest and test_selected_admit_manifest compares the installed manifest with it.
• Red first: the parent's own scripts/test_ci_docs_only.py fails on the parent's tree with Lists differ: ['tools/selected_admit/DESCRIPTOR-V1.md'] != []; the new test file fails on the parent's rule (both pinned tests and the new test of the page) and passes on the new tree (27 tests). Through the rule's command line a change to the page alone is docs-only on the parent and not on the new tree; a documentation page and another Markdown page of the same directory stay docs-only. The new test replays the page in real git histories: an edit, a rename away, a rename onto its path and a deletion all run the proof jobs.
• The eight preflight slices and their cover row pass; the 26 documentation and source guards and the net documentation-revert check end as on the third commit (the native-versions check skips for want of private fixtures). docs/ci.md keeps its length, which the preflight pins.

Not run for this head: the CLOUD verify profiles on x86_64, the parity sweep, any macOS build, and the review of the final head; the train gate runs them once on the train head (the members combined on main), not on this head alone. The hosted CI legs of this pull request run on every push; their results are on its checks page and are not listed here. The manifest test builds its second tree as Release only, so the Debug, kernel and sanitizer profiles of the full suite run it without building the helper in those modes.

🤖 Generated with Claude Code