ci: rebalance native module shards from measured Actions timings - #1148

Scope

Refresh the native module timing weights used by validate's existing greedy planner, and prefer other shards over shard 0 when estimated loads tie. Old weights estimated test_pre_push_gate at 22.5 seconds; three recent successful CI runs put it at a 917.9-second median. Shard 0 also has 111 seconds of measured fixed validation work. The new assignment isolates the two largest modules on shards 1/2 and leaves shard 0 headroom for its validators. The eight-shard matrix, job names, native fixtures and aggregate are unchanged.

Reconcile the ownership-dependent statements in both macOS decisions: live readback identifies a public Organization repository on GitHub Free, so the merge-queue availability reopen triggers are met. Queue activation and a new macOS policy remain separate follow-ups. Unpublished Blacksmith working records are cited by name only, with a scoped Markdown privacy regression.

• Base commit: f165498aa6340860ef52de3a720ca2981eced062
• Lane: lane:foundation
• Owned paths: scripts/validate_shards.py (assignment tie-break only), scripts/validate_shard_weights.json, tests/test_validate_shards.py, tests/test_workflow_hardening.py (hang-fixture shard selection only), the ownership-dependent statements in docs/decisions/2026-10-03-macos-{ci-scope,full-suite-coverage}.md, evidence/artifacts/validate-shard-balance-20261015/{README.md,timings.json}, and their entries in manifests/evidence.json.

Branch history cleanup

The PR branch is squashed to 6028d29eb004b60cfbd729cfbd9ff3f5d75446fa, one commit on its original base f165498aa6340860ef52de3a720ca2981eced062. Its tree is identical to abe6b7c8891554ee424de7f74505a1af94498be8; neither removed unpublished-source digest occurs in the new branch's history.

The replaced commits cc52dc0cd0116ba422b797e688748a5aa368f257, 18412c35666cc12f2d5fba095d36ae5783abf32c, 7a93124d834b310d3c341669f08d0439a6e1a1af, and abe6b7c8891554ee424de7f74505a1af94498be8 may remain reachable by SHA on GitHub until garbage collection. A purge of retained GitHub copies is an owner-only GitHub Support request; this branch rewrite does not perform that purge.

SOTA sources

• Native assignment source: scripts/validate_shards.py at base f165498a, lines 135–160. It uses descending-duration/least-load assignment and preserves native module ownership; ModuleSuite/run_shard retain module fixtures. The assignment's tie-break accounts for the 22 steps guarded by exactly if: matrix.shard == 0: 21 in validate.yml lines 59–239 and the semantic-contract step at lines 379–381, at the same pin. The PR-only condition at line 120 is excluded. Their measured totals are 111/112/99 seconds in the three baseline jobs.
• CPython v3.12.3, Lib/unittest/runner.py, _printDurations, case.py, TestCase.run, and suite.py, TestSuite.run: hosted native duration samples and indivisible fixture semantics.
• GitHub Actions workflow jobs REST API, artifacts REST API, and matrix documentation, accessed 2026-10-15. gh 2.102.0 fetched job/step timestamps, native logs and uploaded inventories from runs 38486609305, 38647310087, and 38675090239.
• Ownership readback, 2026-10-15: gh api orgs/native-agent-stack --jq '{type, plan: .plan.name}' returned {"type":"Organization","plan":"free"}; gh api repos/native-agent-stack/native-agent-stack --jq .owner.type independently returned Organization.
• Current merge-queue eligibility, larger-runner plans, and Actions concurrency limits, accessed 2026-10-15. The repositories endpoint returned private=false; merge queues are available for public organization repositories. The active rules/branches/main types contain no merge_queue, so availability reopens the decision for evaluation without claiming activation. Larger runners still require Team or Enterprise Cloud; a Support concurrency increase remains unverified.
• Privacy custody: correction #161 as recorded in 2026-10-12-harbor-parallel-runtimes.md, lines 119–131 at 7a93124. Unpublished working records are named without digests. The existing JSON guard's Markdown gap is separately recorded; this PR adds a narrow README regression without copying the private digests into the test.

Evidence-class table

MEASURED before / after

Job seconds, completed_at - started_at; queue time is excluded. Before uses the latest successful main push on each sampled day available before 2026-10-15 05:50Z, all attempt 1. Different heads and runner conditions make this an observational comparison, not a controlled paired benchmark. The broader Blacksmith screen's 30-minute median is not reused as this sample's baseline.

After: run 38696923599, complete attempt 2, head 7a93124d834b310d3c341669f08d0439a6e1a1af, tested merge checkout 30e0ca2030b051fe88192914e3e0e2a41662664f. All eight shards and the unchanged aggregate passed: 14,971 tests, zero failures/errors, 1,299 skips, 414 executed modules, complete counts. This was gh run rerun 38696923599 for all jobs, with no code changes.

Attempt 1 is also retained below: seven shards passed, but shard 7 failed the unchanged metrics-registration fixture described below and the aggregate correctly failed. No results are stitched across attempts.

The successful complete rerun measured a 1036-second critical job (17m16s), 183 seconds / 15.0% below the 1219-second before median (20m19s). This is an observed improvement in that run, not evidence of a stable 15% gain: the same head and merge checkout in attempt 1 had a 1606-second critical job (26m46s), 387 seconds / 31.7% worse, with a separate metrics-fixture failure. Both results remain visible.

The indivisible native pre-push suite took 1529.127 seconds in attempt 1 and 968.762 seconds in attempt 2; the latter is still the critical path. Moving module assignments cannot shorten that suite. The isolated landscape-sweep suite took 756.240 and 767.496 seconds respectively. Shard 0's measured job times were 461 and 643 seconds after the tie-break left room for its fixed validators. These observations support the assignment mechanism and expose substantial runtime variability; different before revisions, runner conditions and queue delays prevent a controlled causal claim.

Attempt 1's failed test is tests.test_adaptive_paper_metrics.FileSdRegistrationTests.test_a_clean_stop_of_an_unfinished_trial_stays_registered (finished/not_ready): its exporter registration wait expired after 30 seconds. The test and exporter are unchanged from the base; the isolated test passes locally in 0.94 seconds and the complete attempt-2 rerun passed without any code/test change. A synthetic reuse of a stopped exporter's loopback port reproduced the exact timeout while the exporter was running and its target remained listed: the fixture waits for a target absent from its previous set. That demonstrates a pre-existing fixture mechanism, not proof that CI reused the port. No trading code or assertions are changed here. Both repaired hang-diagnostic tests passed in native shard 4.

Expected balance and limits

The frozen CI timing table predicts old assignment loads of [435.534, 356.046, 1033.212, 370.260, 1180.264, 340.245, 325.995, 391.078] seconds and refreshed loads of [456.485, 917.878, 775.843, 456.485, 456.485, 456.485, 456.485, 456.485]. These are historical scheduling estimates, separate from the measured table and not a guarantee of current module runtimes.

The regression asserts (a) maximum expected shard time <= the largest indivisible module + 1 second, (b) every other shard <= that largest module, and (c) max/median < 2.1. Old weights fail all three assertions (ratio 3.1005); updated weights pass (ratio 2.011). An additional assertion includes the median 111-second shard-0-only validator cost: the original tie-break fails at 1028.878 seconds against 918.878 allowed; the new tie-break gives shard 0 567.485 seconds including its validators, leaving the critical path at the largest module. No bound was weakened. The one-second allowance covers numeric/proxy noise, not common runner setup.

The duplicate-receipt fixture now duplicates whichever shard contains the test, rather than assuming shard 0. It still requires aggregate rejection and zero counted tests, and now asserts the exact unknown-shard list.

The hang-diagnostic fixtures similarly locate their single test through the native assignment. The old hardcoded shard 0 became empty after the tie-break change: both fixtures failed locally and in job 116287596097, run 38694437548 at 18412c35, returning 0 instead of 124. The fix changes only fixture shard selection. The hang body, 3-second timeout, signal, status 124 and traceback assertions are unchanged; both diagnostic test methods are AST-identical before/after. The completed after column uses the subsequent fixed head's complete workflow.

The baseline medians are 917.878 seconds / 15.30 minutes for test_pre_push_gate and 775.843 seconds / 12.93 minutes for test_landscape_sweep_harness. While those fixtures remain indivisible, balancing the six remaining shards (456.485 seconds each) cannot produce near-equal durations or max/median < 1.5. Setup, discovery, aggregation, queues, PR-only steps and new modules using the existing 10-second fallback are outside the frozen model; the additional assertion accounts separately for the shard-0-only validators.

All 24 extracted event counts match their native test counts; reconstructed suite times agree within 0.37 seconds. Module totals remain proxies, incorporating fixture/log-delivery effects. Of 410 owners, 392 have three observations, 16 have two, and two have one. Unknown modules still execute via the existing fallback.

Local commands run

python3 -m unittest tests.test_validate_shards.PlannerProperties.test_timing_readme_cites_unpublished_records_without_digests -v
  before privacy correction at 7a93124: exit 1, README digest invariant fails without echoing a digest
python3 -m unittest tests.test_validate_shards -v
  after privacy correction: exit 0, all 15 tests pass
python3 -m unittest tests.test_validate_shards.PlannerProperties.test_committed_ci_timings_bound_eight_shard_balance -v
  before refresh: exit 1, all three balance assertions fail
python3 -m unittest tests.test_validate_shards -v
  after refresh: exit 0, 14 tests pass
python3 -m unittest tests.test_validate_shards.PlannerProperties.test_committed_ci_timings_bound_eight_shard_balance -v
  with refreshed weights but old tie-break: exit 1 on the new fixed-overhead assertion
  with the new tie-break: exit 0; all original bounds remain unchanged
python3 -m unittest tests.test_workflow_hardening.ValidateShardWorkflowContract tests.test_workflow_hardening.ValidateShardFinalStepRuns -v
  exit 0, 10 existing matrix/aggregate contract tests pass
python3 -m unittest tests.test_workflow_hardening.ValidateSuiteStepTracesAHang -v
  before fixture correction at 18412c35: exit 1, both tests fail (empty shard, 0 instead of 124)
  after correction: exit 0, both original diagnostic oracles pass
python3 -m unittest tests.test_workflow_hardening tests.test_validate_shards -v
  exit 0, 132 tests, 1 existing skip; also ran the two fixtures plus the 24 focused contracts successfully
python3 scripts/validate.py
  exit 0 before each commit: 71 components, 12049 hashed files, 4 profiles, 240 receipts
git diff --check
  exit 0

Decision record

evidence/artifacts/validate-shard-balance-20261015/README.md records sources, the layer-resolution delta, extraction, before timings, attainable bound and follow-up. It retains the existing native module design; hosted results that do not improve the critical path reopen the estimates.

Follow-ups

GPT fix2 delta P3s (cc-reads-20261005/fix2-pr1148/GPT-VERDICT-6028d29eb.md): broaden the planner citation to support its full explanation; repair inherited registry hygiene; retain captures for the fixed-validator measurement; and add a fail-first regression for the organization-eligibility correction. The local review's P2 identifies the same missing regression at docs/decisions/2026-10-03-macos-ci-scope.md:77–81; address it in a follow-up with an observed failure against the prior ownership text. These are recorded follow-ups, with no additional code change in this PR.

CC delta P3: reconcile docs/decisions/2026-10-03-macos-full-suite-coverage.md:120–127 with evidence/receipts/github-ci-measurements-20261003.json:153–158. The corrected decision removed the historical 2026-10-03 owner-type quote that the receipt still says the decision carries. Preserve the historical observation and distinguish it from current eligibility when repairing that provenance reference in a follow-up.

Extend the general published-digest guard to Markdown in a separate PR; the current JSON-only guard does not catch this format. The scoped README invariant here remains in place. Re-evaluate the native merge-queue option now that the documented public-organization availability trigger is met, including required merge_group checks and ruleset integration before activation.

Fixture-aware splitting remains separate. The three baseline pre_push_gate module estimates range from 684.525 to 1171.164 seconds, median 917.878. The first hosted measurement in this PR (run 38690547118, head cc52dc0cd0116ba422b797e688748a5aa368f257, before the tie-break correction) passed all eight shards and the aggregate, but its pre-push module alone took 1480.146 seconds / 24.67 minutes. Its job times were [1671,678,482,470,539,538,580,484] seconds: that run did not improve the 1219-second before critical-path median, and disproves treating the historical 15.30-minute estimate as a current runtime ceiling. The final head's own run is reported in the after column; no wall-time speedup is inferred from model predictions. Native fixture-preserving assignment cannot reduce an indivisible module's execution time.

Refresh timing inputs when CI provides changed evidence. Full workflow reruns remain the practice; a partial shard rerun cannot satisfy the unchanged eight-report aggregate.

Host evidence

Not applicable: no files under evidence/hosts/ or platform-status claims change.

Checklist

• No new or changed GitHub Actions pins or workflows; eight matrix names and aggregate preserved.
• Existing workflow permissions preserved.
• No secrets printed or committed; no new secret required.
• No new paid hosting, subscription or billing surface.
• Peer-owned files and worktrees preserved.
• No new scripts/*.py or workflow files; no new source-policy grant needed.
• Repository no-reply identity verified before and after commit; push uses an explicit lease.
• This PR's own complete successful CI run and eight job times recorded above; the unsuccessful prior attempt remains separately labeled.