Skip to content

perf(ci): recalibrate compact shard hints from 7-run loaded-fleet means#110172

Merged
steipete merged 1 commit into
mainfrom
perf/ci-shard-recal-2
Jul 17, 2026
Merged

perf(ci): recalibrate compact shard hints from 7-run loaded-fleet means#110172
steipete merged 1 commit into
mainfrom
perf/ci-shard-recal-2

Conversation

@steipete

@steipete steipete commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

After the whale-import fix (#109772) and the first calibration pass, compact shard bins drifted again: measured large-bin walls reached 389s with a predicted max/mean of 1.38 — the shard envelope is the run-level floor for every test-touching PR.

Why This Change Was Made

Second calibration pass from a 7-run dataset (147 compact job logs across green runs whose heads include the whale-fix hints; one stale-base run discarded; outliers outside [median/1.5, median×1.5] dropped): all 82 hinted groups re-measured to loaded-fleet means. Predicted bin walls go from 220/301/200/219/257/112 (max 301, max/mean 1.38) to 235/232/229/228/228/156 (max 235, max/mean 1.08) at the same 22-job count — a ~22% ceiling cut.

COMPACT_NODE_TEST_JOB_SECONDS 220→235: with honest loaded-fleet hints, 220 adds one job per pool for zero ceiling win (sweep-verified at 220/225/230/235/240); measured setup overhead is 60s median, so 235s bins land near the 5-minute PR budget. Exclusive cap, stripe counts, pinned-group regex, and group-count cap untouched; STRIPE_FILE_SECONDS_HINTS deliberately unchanged (stripe walls sit in exclusive bins far below the ceiling; rationale in the commit body). Hints-only diff — no packer logic changes.

User Impact

None at runtime — CI-only. Test-touching PRs: shard ceiling 301→236s predicted, tightening the run-level floor by ~1min.

Evidence

  • Data provenance (7 run ids) and full before/after hint tables in the commit body.
  • Plan tests green: ci-node-test-plan (20), ci-run-node-test-shard + ci-changed-node-test-plan (30), ci-workflow-guards (80 passed/1 skipped); rerun green after rebase onto current main.
  • oxfmt + git diff --check clean; autoreview (codex gpt-5.6-sol) first-pass clean: "patch is correct (0.91)", zero findings.
  • Live proof: this PR's own compact shards run under the new plan.

@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts size: S maintainer Maintainer-authored PR labels Jul 17, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 17, 2026
@clawsweeper

clawsweeper Bot commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 17, 2026, 7:23 PM ET / 23:23 UTC.

Summary
The branch replaces compact Node-test shard duration estimates with seven-run loaded-fleet measurements and changes the compact per-job packing cap from 220 to 235 seconds.

PR surface: Other +3. Total +3 across 1 file.

Reproducibility: not applicable. the PR tunes CI shard planning rather than reporting a user-reproducible product defect. The relevant validation is completed exact-head planner and shard timing evidence.

Review metrics: none identified.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦐 gold shrimp
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Attach or link redacted completed exact-head compact large and small shard timing output.
  • Have a maintainer decide whether that run confirms the seven-run calibration is representative.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR describes a live exact-head compact-shard run, but the available material does not expose completed plan-and-timing output that directly demonstrates the claimed after-fix balance. Attach redacted completed CI timing output; after updating the PR body, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.

Risk before merge

  • [P1] The values alter CI scheduling rather than product behavior, but the claimed lower run-level ceiling remains unverified until the in-progress exact-head compact shard jobs publish completed timing output.
  • [P1] The maintainer label requires explicit human handling even if CI completes green.

Maintainer options:

  1. Verify the exact-head shard result first (recommended)
    Wait for the in-progress compact large and small shard jobs, then compare their completed timing output with the claimed job-count and ceiling improvement before merge.
  2. Keep the existing calibration
    Pause this tuning change if the completed run does not support the new loaded-fleet estimates or indicates a CI scheduling regression.

Next step before merge

  • [P1] The remaining action is operational evidence acceptance and protected-label maintainer judgment, not a narrow automated repair.

Maintainer decision needed

  • Question: After the exact-head compact shard jobs complete, should the measured timing output be accepted as sufficient evidence that the 235-second cap preserves job count while improving balance?
  • Rationale: The patch intentionally tunes operational CI scheduling from fleet measurements; whether the new measurement set is representative enough to change the production CI plan is a maintainer-owned operational judgment.
  • Likely owner: steipete — They have the clearest merged-history connection to the preceding timing-profile change and supplied the calibration methodology.
  • Options:
    • Accept completed timing evidence (recommended): Merge after the completed exact-head large and small compact shards support the claimed balance and no new queue or timeout regression is visible.
    • Recalibrate from a broader sample: Keep the existing planner values and request another measurement window if the completed run materially disagrees with the seven-run model.

Security
Cleared: The proposed diff is confined to CI duration estimates and a packing cap; no dependency, permission, secret, supply-chain, or runtime security boundary change is evident.

Review details

Best possible solution:

Retain the hints-only approach if completed exact-head compact shard output confirms the predicted job count and materially flatter wall-time envelope, then land it through normal maintainer review.

Do we have a high-confidence way to reproduce the issue?

Not applicable: the PR tunes CI shard planning rather than reporting a user-reproducible product defect. The relevant validation is completed exact-head planner and shard timing evidence.

Is this the best way to solve the issue?

Unclear: recalibrating the existing planner is the narrowest implementation path, but the available context does not yet expose completed exact-head timing output needed to confirm that these particular values are the best calibration.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against a4287653375b.

Label changes

Label justifications:

  • P3: This is a CI scheduling optimization with no reported product-runtime regression.
  • merge-risk: 🚨 automation: The patch changes compact test-shard packing estimates and the per-job cap, which can affect CI duration and runner scheduling after merge.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR describes a live exact-head compact-shard run, but the available material does not expose completed plan-and-timing output that directly demonstrates the claimed after-fix balance. Attach redacted completed CI timing output; after updating the PR body, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Other +3. Total +3 across 1 file.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 1 90 87 +3
Total 1 90 87 +3

What I checked:

  • Changed planner surface: The proposed patch changes only compact CI planning data and its packing cap; its comments state that the estimates are advisory and affect balance rather than test correctness. (scripts/lib/ci-node-test-plan.mjs:33, c754cf4ee877)
  • Prior review continuity: The prior completed ClawSweeper cycle found no code defect but required inspectable exact-head shard output; the current context shows the branch was force-pushed afterward, while the relevant large and small compact shard checks remain in progress. (c754cf4ee877)
  • Adjacent merged provenance: The calibration explicitly follows the merged CLI-runner import-cost improvement, which changed the timing profile but did not itself implement this planner recalibration. (scripts/lib/ci-node-test-plan.mjs:33, f095549d1d31)
  • Repository-policy inspection limitation: A read-only command intended to read the complete on-disk policy, inspect status, and inspect the selected skills was blocked by the sandbox before execution; this review therefore does not claim a full on-disk policy or source-history pass. (AGENTS.md)

Likely related people:

  • steipete: They authored the related merged CLI-runner performance change and this follow-on calibration, establishing a concrete merged-history connection to the compact CI timing surface. (role: recent CI performance contributor; confidence: medium; commits: f095549d1d31, c754cf4ee877; files: scripts/lib/ci-node-test-plan.mjs, src/agents/cli-runner.reliability.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-07-17T21:03:08.374Z sha 32102db :: needs real behavior proof before merge. :: none

@steipete
perf(ci): recalibrate compact shard hints to post-whale loaded-fleet …
c754cf4 
…walls

Second calibration pass after the cli-runner reliability whale fix
(#109772) and the stripe-wall correction (#109899): the compact group
hints drifted below loaded-fleet reality, so FFD packed the heaviest
groups into ~300s bins (checks-node-compact-large-2 at 242-314s shard
wall, up to 389s job wall) while tail bins idled.

Method: parsed [shard:*] begin/end timestamps from all 147 compact job
logs across seven green CI runs whose head SHAs carry the whale-fix
hints (runs 29605136624, 29605203485, 29605983019, 29606701461,
29611308972, 29611457693, 29611500865; 7 samples per group). Each hint
is the per-group mean after dropping cache-warm/contention outliers
outside [median/1.5, median*1.5].

Packing cap: with honest loaded walls a 220s cap no longer fits either
pool (adds one job to each); 235s keeps the same job count (6 large +
15 small + dist) and flattens the ceiling. Median job setup overhead
measured 60s (p90 87s), so a 235s bin stays near the 5-minute PR
budget.

Predicted bin shard walls (sum of measured group means):
  before: large 220/301/200/219/257/112, small heavies 251/235/218/217/215, max 301
  after:  large 235/232/229/228/228/156, small heavies 236/234/234/234/233, max 236
  large-pool max/mean 1.38 -> 1.08

Biggest hint deltas (old -> new): startup-core 98->156, core-tooling-4
71->125, core-unit-fast-isolated 50->90, core-tooling-3 82->108,
runtime-server 14->29, storage-state 55->70, media-ui 113->124,
tui-pty 103->116, runner-cli-1 18->8, unit-src-security 108->95.

Exclusive bins keep their 150s cap and 5-bin count: tui-pty (116) no
longer shares a bin with tooling-isolated (166s combined measured);
FFD now pairs tooling-2 with tooling-isolated at 144s.

STRIPE_FILE_SECONDS_HINTS left unchanged: measured stripe walls sit in
exclusive bins far below the packing ceiling (tooling stripes
94-125s), and refitting per-file hints would reshuffle stripe
membership and invalidate the measured group means above.
@steipete
steipete force-pushed the perf/ci-shard-recal-2 branch from 32102db to c754cf4 Compare July 17, 2026 23:19
@steipete
steipete merged commit f0f06ea into main Jul 17, 2026
96 checks passed
@steipete
steipete deleted the perf/ci-shard-recal-2 branch July 17, 2026 23:48
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 18, 2026
@steipete
perf(ci): recalibrate compact shard hints to post-whale loaded-fleet …
8b7f8de 
…walls (openclaw#110172)

Second calibration pass after the cli-runner reliability whale fix
(openclaw#109772) and the stripe-wall correction (openclaw#109899): the compact group
hints drifted below loaded-fleet reality, so FFD packed the heaviest
groups into ~300s bins (checks-node-compact-large-2 at 242-314s shard
wall, up to 389s job wall) while tail bins idled.

Method: parsed [shard:*] begin/end timestamps from all 147 compact job
logs across seven green CI runs whose head SHAs carry the whale-fix
hints (runs 29605136624, 29605203485, 29605983019, 29606701461,
29611308972, 29611457693, 29611500865; 7 samples per group). Each hint
is the per-group mean after dropping cache-warm/contention outliers
outside [median/1.5, median*1.5].

Packing cap: with honest loaded walls a 220s cap no longer fits either
pool (adds one job to each); 235s keeps the same job count (6 large +
15 small + dist) and flattens the ceiling. Median job setup overhead
measured 60s (p90 87s), so a 235s bin stays near the 5-minute PR
budget.

Predicted bin shard walls (sum of measured group means):
  before: large 220/301/200/219/257/112, small heavies 251/235/218/217/215, max 301
  after:  large 235/232/229/228/228/156, small heavies 236/234/234/234/233, max 236
  large-pool max/mean 1.38 -> 1.08

Biggest hint deltas (old -> new): startup-core 98->156, core-tooling-4
71->125, core-unit-fast-isolated 50->90, core-tooling-3 82->108,
runtime-server 14->29, storage-state 55->70, media-ui 113->124,
tui-pty 103->116, runner-cli-1 18->8, unit-src-security 108->95.

Exclusive bins keep their 150s cap and 5-bin count: tui-pty (116) no
longer shares a bin with tooling-isolated (166s combined measured);
FFD now pairs tooling-2 with tooling-isolated at 144s.

STRIPE_FILE_SECONDS_HINTS left unchanged: measured stripe walls sit in
exclusive bins far below the packing ceiling (tooling stripes
94-125s), and refitting per-file hints would reshuffle stripe
membership and invalidate the measured group means above.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. scripts Repository scripts size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@steipete