Skip to content

ci: split the long test rows, shard the full CI legs and run them on the larger hosted runners - #544

Merged
luisleo526 merged 6 commits into
mainfrom
ci/383-proof
Oct 27, 2026
Merged

luisleo526 merged 6 commits into
mainfrom
ci/383-proof

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Summary

A post-merge CI round on main took 94 minutes (ubuntu Release 74, native 74, live-sanitizers 91, live-tsan 64, kernel 58, sanitizers 52, Debug 42). Almost all of that is ctest wall: each leg ran its ~1,240 rows on half of a shared 16-core VM, and a handful of single rows of 8 to 65 minutes set the critical path. This change splits those rows into parallel rows that run exactly the same cases, lets ci_verify.py shard the native-live profiles like the other profiles, and moves the legs onto the larger hosted runners as shard jobs. It also carries a separate, self-contained test fix (the flat-RSS sink test, last section).

What changes

Test rows (each: K slice rows plus one public cover row that proves, from the program's own enumeration, that the union of the slices is the old row's case set with no overlap; the old row name is gone; labels, TMPDIR and conditions are kept):

Old row Measured (Release, serial) Now
test_ci_verify 3,911 s 8 unittest slices (lab: 150-397 s each)
test_ci_preflight 963 s 8 unittest slices (lab: 20-58 s each)
test_native_command_after 1,916 s (72 seeded books) 8 slices; one book alone takes 563 s on one core and has a slice of its own; the other heavy books are spread; COST orders the start (it is no time limit)
test_open_report_rows_flat 1,080 s 4 slices; each 100000-lot branch stays whole so its ordered fold is not split
native_live_equivalence_e2e 3,334 s of the live-sanitizers ctest one row per probe (8 rows)

The registered-row floors follow (+28 rows in every engine profile, +8 more in native; RELEASE_MIN_TESTS 938, KERNEL_MIN_TESTS 365, live floor 21), and PR_SLOW_ROWS pins the 43 literal slow names as an ordered tuple.

ci_verify.py: --ctest-shard I/N is accepted for the native, live-sanitizers and live-tsan profiles; the union validator and the shared-stage rules cover them.

Workflows:

  • ci-post.yml: Release 4 shard jobs and Debug 2 on pf-linux-x64-16, kernel-only 2 on pf-linux-x64-16, sanitizers 4 on pf-linux-x64-32. Aggregate jobs keep the old check names (build (ubuntu-24.04, Release), build (ubuntu-24.04, Debug), sanitizers, kernel-only), so the release gate's required legs do not change, and they skip together with their shards on a classified docs-only change.
  • native-live.yml: live-sanitizers and live-tsan on pf-linux-x64-64 (not sharded: every shard would rebuild the instrumented tree); native as 4 shard jobs on pf-linux-x64-16 with an aggregate native-live (native) that needs every shard and its proof output.
  • scripts/ci_preflight.py, its tests and actionlint.yaml: exact runner pin per job, the labels refused for every other job, the shard list 1..N equal to the --ctest-shard denominator, the aggregates and the proofs; no timeout-minutes anywhere.

Docs: docs/ci.md and the pages that cite its floor table are updated; docs/ci.md keeps its line count.

Separate: flat-RSS sink test (docker/pineforge_sink_finalize_test.py only)

The flat-RSS test judged one tiny and one 64 MiB probe per mode against a 1 MiB bound and failed on the scatter of the tiny baseline. It now runs three fresh-process probes of each size per mode and decides by min(big) - max(tiny) in the pure function rss_flatness_verdict; the bounds are unchanged. A planted-retention row (2 MiB kept, file and stdout mode) and a positive control with the same launcher and no retention show that a launcher reporting a zero peak or measuring the wrong process cannot pass.

Evidence

  • Full lab preflight (59 stages) passed on the tree of this change before the last rebase onto main; the lab ctest runs of the slice rows passed with the times in the table above.
  • This is a draft until the engine legs of the final tree are green on a lab host and the per-leg walls of the dispatched ci-post and native-live runs are measured; the walls and the executed-test-name comparison against main are posted here as a comment, and the Summary figures are corrected to them if they differ.

Risk and rollback

The shard jobs build the whole tree each; the 15-minute target holds only if build plus the longest shard fits, which the dispatch measures. Rollback: revert this commit (the old rows, the unsharded legs and the VM pins return together).

🤖 Generated with Claude Code

…the larger hosted runners

A post-merge round of CI on main 2cd229f took 94 minutes (ubuntu Release 74,
native 74, live-sanitizers 91, live-tsan 64, kernel 58, sanitizers 52, Debug 42)
because each leg ran its ~1,240 ctest rows on half of a shared 16-core VM, with
a handful of single rows of 8 to 65 minutes setting the critical path. This
change splits those rows into parallel rows that run exactly the same cases,
lets ci_verify.py spread the native-live profiles over shard jobs like the other
profiles, and moves the legs to the larger hosted runners.

Test rows (each: K slice rows + one public cover row that proves, from the
program's own enumeration, that the union of the slices is the old row's case set
with no overlap; the old row name is gone; labels, TMPDIR and conditions are kept)
- test_ci_verify (3,911 s on the Release leg): 8 unittest slices.
- test_ci_preflight (963 s): 8 unittest slices (lab: 20 to 58 s each).
- test_native_command_after (1,916 s, 72 seeded books): 8 slices; one book alone
  takes 563 s on one core (live 60, lower path, full window, calculate-on-fills)
  and has a slice of its own; the other heavy books are spread; the rows that
  hold them carry a COST hint so ctest starts them first (COST orders the start,
  it is no time limit).
- test_open_report_rows_flat (1,080 s): 4 slices; each 100000-lot branch stays
  whole so its ordered fold is not split (lab: two slices of 493 s).
- native_live_equivalence_e2e (3,334 s of the live-sanitizers ctest): one row per
  probe (8 rows) through the existing adapter, which forwards --probes.
- The registered-row floors follow: +28 rows in every engine profile, +8 more for
  the equivalence rows in native: RELEASE_MIN_TESTS 938, KERNEL_MIN_TESTS 365,
  absent-fixture floors 631 and 334, EXCLUDED_REGISTERED_MIN debug 904,
  sanitizers 895, native 1000, release 923, kernel 365, live floor 21 (20 without
  transport); PR_SLOW_ROWS pins the 43 literal slow names as an ordered tuple.

ci_verify.py
- --ctest-shard I/N is accepted for the native, live-sanitizers and live-tsan
  profiles; the union validator and the shared-stage rules cover them.

Workflows
- ci-post.yml: Release runs as 4 shard jobs and Debug as 2 on pf-linux-x64-16,
  kernel-only as 2 on pf-linux-x64-16, sanitizers as 4 on pf-linux-x64-32;
  aggregate jobs keep the old check names so the release gate's required legs do
  not change, and skip together with the shards on a classified docs-only change.
- native-live.yml: live-sanitizers and live-tsan on pf-linux-x64-64 (not sharded:
  every shard would rebuild the instrumented tree); native as 4 shard jobs on
  pf-linux-x64-16 with an aggregate native-live (native) that needs every shard.
- ci_preflight.py, its tests and actionlint.yaml: an exact runner pin per job, the
  labels refused for every other job, the shard list 1..N with N equal to the
  --ctest-shard denominator, the aggregates and the proofs.

Flat-RSS sink test (docker/pineforge_sink_finalize_test.py only)
- The flat-RSS test judged one tiny and one 64 MiB probe per mode against a 1 MiB
  bound and failed on the scatter of the tiny baseline. It now runs three
  fresh-process probes of each size per mode and decides by min(big) - max(tiny)
  in the pure function rss_flatness_verdict; the bounds are unchanged (a 64 MiB
  peak may exceed the tiny peak by at most 1 MiB and must stay within the tiny
  baseline plus 2 MiB). A planted-retention row (2 MiB kept, file and stdout mode)
  and a positive control with the same launcher and no retention show that a
  launcher reporting a zero peak or measuring the wrong process cannot pass; the
  rows record rss_headroom_to_bound_bytes and rss_retention_planted.

docs/ci.md and the pages that cite its floor table are updated (docs/ci.md keeps
its line count). The sentences that leave were added by 91353de (the
registration constants and floors, the undivided minTests, the 29-row slow
inventory and 41 source-labelled names), dad2530 (the single equivalence row and
the native floor), 0212cab (the full-run floors, the slow label and
EXCLUDED_REGISTERED_MIN paragraphs), b0f4341 (the post-fleet routing and
runner-split paragraphs), 6c95d82 (the amd64 and arm64 cell sentences),
065ef1d (the three native-live matrix cells), 3a0d150 (the native-live row of
the runner table and the larger-runner label sentence), e11d9fc (the release and
kernel rows of the floor table), b97c409 (the two numbers are constants of the
profile) and d093560 (the 29 CTest rows labelled slow).

Not run on the author's machine.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Comment thread scripts/test_ci_verify.py
self.assertNotIn('-L', argv)
self.assertEqual(sorted(flat), sorted(inventory))
self.assertEqual(len(flat), len(set(flat)))
self.assertIsNone(ci_verify.validate_ctest_shard_union(summaries))
Comment thread scripts/test_ci_verify.py
with self.subTest(profile=profile):
summaries = self.shard_summaries(
1, profile=profile, inventory=self.native_live_inventory(), labelled=())
self.assertIsNone(ci_verify.validate_ctest_shard_union(summaries))
Comment thread scripts/test_ci_verify.py
self.assertEqual({stage['name'] for stage in summary['stages']
if stage['status'] == 'deferred'},
shared if index != 1 else set())
self.assertIsNone(ci_verify.validate_ctest_shard_union(baseline))
Comment thread scripts/test_ci_verify.py
self.assertIn(f'below the floor of {floor}', driver.failures[0]['error'])
summaries = self.shard_summaries(
4, profile=profile, inventory=self.native_live_inventory(), labelled=())
self.assertIsNone(ci_verify.validate_ctest_shard_union(summaries))
Comment thread scripts/test_ci_verify.py
with self.subTest(profile=profile):
baseline = self.shard_summaries(
4, profile=profile, inventory=inventory, labelled=(), ctest_skipped=(pinned,))
self.assertIsNone(ci_verify.validate_ctest_shard_union(baseline))
Comment thread scripts/test_ci_verify.py
with self.subTest(profile=profile):
baseline = self.shard_summaries(
1, profile=profile, inventory=inventory, labelled=(), ctest_skipped=(pinned,))
self.assertIsNone(ci_verify.validate_ctest_shard_union(baseline))
ci-cx and others added 5 commits October 27, 2026 09:03
…he sliced test rows

The sliced rows (K literal slice rows running one program, plus a public cover
row) made six ctest rows fail on every full engine profile:

- test_private_fixture_reads and its controls: tests/private_fixture_reads.evidence
  cited tests/CMakeLists.txt lines that moved by 27; every citation in the
  record now names the line where its statement sits.
- test_two_world_registration and its controls: the sweep's model of
  pineforge_add_unit_test registered one same-named row per call and skipped the
  function body, so the slice rows were unknown and the slow-row guard fired.
  The model now records function definitions, binds the call arguments, follows
  the STREQUAL branch and reads literal or expanded add_test names. A reached
  CMake error now prints its expanded message; a deleted slice row is named.
- test_cr150_cmake_d: every compiled add_test name needed its own .cpp. A name
  of the form <program>_slice_<i> is now accepted only when i is in 1..K, the
  program's .cpp exists and defines the --slice parser and dispatch, and the
  command is exactly <program> --slice I/K. Plain unknown names still need
  their own source.
- test_native_command_after_slices_cover: the branch pattern expected the
  closing parenthesis right after the first exclusion of the helper's elseif;
  it now accepts any number of further literal exclusions of the same argument.

New controls, each red on the old checker: argument branches and literal rows
in the registration model, a deleted real slice row, six shared-source slice
cases (valid rows, missing program, index outside 1..K, plain unknown name,
wrong command, program without parser or dispatch), and compound exclusions in
the cover pattern. No registration, floor, label or doc line changes.

Not run on the author's machine.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…over at its own slice row

Two mutations of the command_after cover replaced the first "--slice 8/8" in
tests/CMakeLists.txt. The first occurrence belongs to the preflight slice
row, so the text changed but the parsed command_after registration did not,
and the cover reported no error for mutations it was written to catch. Both
mutations now replace the argument inside the command_after slice 8 row the
test has already located, so the overlap case gives that row selector 7/8 and
the zero-case row selector 9/8, and the cover reports "slice assignments
overlap" and "empty or out-of-range slice" as the assertions expect. No
checker logic, assertion or other mutation changes.

Not run on the author's machine.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ce parent into its probe rows

tests/private_fixture_rows.txt pins the parent row native_live_equivalence_e2e
and scripts/ci_verify.py expands it into one name per probe of the driver's
literal PROBES dictionary after checking the raw 376-entry count. The read
sweep registered the eight per-probe rows from the runner loop but compared
the raw parent against them, so it reported the parent as pinned without a
registration on every full engine profile.

The sweep now reads the driver's single literal PROBES dictionary, validates
the slugs (lower-case, hyphenated, distinct), and compares the expanded names
with the registrations, each keeping the parent pin's line. The raw pin file
and its count are unchanged. A malformed or duplicated PROBES declaration
exits 2 with a diagnostic; every other pin keeps its exact treatment.

New controls, each red on the old sweep: a probe removed from the runner loop
and a driver probe the loop lacks are each reported by the missing child's
name, and a duplicated driver probe is rejected.

Not run on the author's machine.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… the tiny-probe scatter

The planted-retention control kept 2 MiB in the big probe and expected the
flatness verdict to refuse it. On a lab host the plant showed up as 1.75 MiB
of peak growth (7/8 of the plant) and one tiny probe scattered up by 0.875
MiB, so min(big) - max(tiny) was 917504 bytes, below the 1 MiB bound, and the
verdict called a retaining launcher flat.

The plant is now a named constant, RSS_PLANTED_RETENTION_BYTES, twice the
2 MiB baseline headroom (4 MiB). At the observed 7/8 ratio it grows the peak
by 3.5 MiB; after a 2 MiB scatter budget 1.5 MiB remains, above the 1 MiB
bound, and with the recorded 917504-byte scatter 2.625 MiB remains. Two pure
rows assert that arithmetic from the constants and the recorded peaks, so the
plant cannot shrink silently. The verdict, the bounds, the probe count, the
stream size, both modes and the positive control with no retention are
unchanged; the control test lost the "2_mib" from its name.

Not run on the author's machine.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the docs-only commits since the branch's base (supported platforms
page, memory-management page re-read). No conflicts.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526
luisleo526 marked this pull request as ready for review October 27, 2026 05:56
@luisleo526
luisleo526 merged commit 3ea5a0d into main Oct 27, 2026
18 of 19 checks passed
luisleo526 added a commit that referenced this pull request Oct 27, 2026
…62 pinned names)

Both the report-memory change and the paged array store added one pinned slow
row each on top of 60, and git merged the two identical edits of the count
into 61. The pinned set now holds 62 names: 43 from the PINEFORGE_PR_SLOW_TESTS
loop and 19 direct CMake LABELS assignments.

The test assertion on len(PINNED_SLOW_TEST_ROWS) and the four sentences in
docs/ci.md that state the count and its split (introduced by 3ea5a0d, ci:
split the long test rows, shard the full CI legs and run them on the larger
hosted runners, #544, and rewritten by the report-memory change) are updated
to 62 and 43 + 19. No other text is removed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 27, 2026
…it (resident, disk, jailed arena) (#556)

memory: state the report-memory bound exactly and add the proofs for it (resident, disk, jailed arena)

The memory-management page said the default sink route keeps the equity
curve out of memory, but not what still grows with the bar count, and that
a jailed run without --report-sink-fd has no sink. It now states, with
sources: the resident equity-report buffers are bounded on the default
unjailed sink route; the curve scratch costs 24 B per script bar on disk
(6 MiB at 262,144 and 48 MiB at 2,097,152 bars) with the E output and the
JSON artifact; in a jailed run the scratch is the RAM-backed arena, so the
same 24 B per script bar are memory there, sized by the hosting wrapper and
failing closed with arena_exhausted; sink-off callers keep the native curve
plus an owning report copy. "O(1) in bars" is not claimed for the jailed
arena or the sink-off path.

Removed text, named. The page text below was introduced by 6d2bf95 (docs:
add the memory-management page, #531) and 30e9f9a (docs: memory-management
page re-read against main after the 1.8.0 changes, #546):
- the section "Use the report sink for bounded memory" is replaced by
  "Report memory: resident buffers, disk and the jailed arena" (its line
  citations were re-read and corrected);
- the phrase "a bounded default route is planned, with no release date" and
  the sentence "Install a report sink; no release date for the default
  route" are removed from the report-memory table (the page documents the
  route as it is);
- the two sink rows of that table are reworded to separate resident buffers
  from disk and arena growth; the sentences about the heap bound of the
  equity curve and the ledger, the sink file growth, and the legacy FULL sink
  route are rewritten into the new section, and the release container's
  default profile sentence of 30e9f9a is replaced by the route table.

Tests: a two-size resident proof for the report scratch curve (N = 262,144
and 8N script bars, RSS growth within 1 MiB, scratch growth exactly 24 B per
added bar, metric block bit-equal to the sink-off result); a corpus
projection test that the temporary-sink route and the sink-off route give
bit-identical metrics and diagnostics on real modules (it replaces the
named-sink summary comparison on corpus modules that this test did before,
and still skips when its manifest variable is unset). Both report-memory
rows are pinned as slow rows (61 pinned names) and registered before the
CR-150 include block. The binary-route peak-RSS row is left out of this
change: its 9-minute wall exceeds the CI leg budget.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

ci: count the array-growth slow row with the report-memory slow row (62 pinned names)

Both the report-memory change and the paged array store added one pinned slow
row each on top of 60, and git merged the two identical edits of the count
into 61. The pinned set now holds 62 names: 43 from the PINEFORGE_PR_SLOW_TESTS
loop and 19 direct CMake LABELS assignments.

The test assertion on len(PINNED_SLOW_TEST_ROWS) and the four sentences in
docs/ci.md that state the count and its split (introduced by 3ea5a0d, ci:
split the long test rows, shard the full CI legs and run them on the larger
hosted runners, #544, and rewritten by the report-memory change) are updated
to 62 and 43 + 19. No other text is removed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

ci: update the slow-row count literals in test_ci_preflight to 62, 43 and 19

test_documented_slow_row_counts_match_literal_population pins the three
numbers that docs/ci.md states for the pinned slow rows (all, from the
PINEFORGE_PR_SLOW_TESTS loop, direct LABELS assignments) and two sentences that
repeat them. The previous commit moved the documented count from 61 to 62
(43 + 19) but left this test at 61, 43 and 18, so test_ci_preflight_slice_6
failed on every profile. The test now states 62, 43 and 19. Only those literals
change.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

memory: move three docker/run_json.py citations after the bulk equity copy

The bulk path-sink equity copy added code above the report-sink helpers in
docker/run_json.py, so the three line ranges the memory-management page cites
for _ReportSink, _build_sink_report and _prepare_stream_report no longer
contained their symbols (doc-anchors: SYMMISS). The ranges now read 6183-6220,
7102-7148 and 6983-7003. Citation numbers only: no sentence is added, removed
or reworded.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>


Also rewritten: the 60-name slow-row counts in docs/ci.md introduced by 3ea5a0d (ci: split the long test rows ..., #544), now 62 = 43 + 19 (the report-memory row and #547's array-growth row); the Pine product host row of the report-memory table in docs/pages/memory-management.md introduced by 27cd77b (engine 1.8.0: cohort tables ..., #541), reworded to separate resident buffers from disk and arena growth.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant