Skip to content

perf(test): fix env-dependent failures, stop verifier lazy-fetch, balance CI stripes#109332

Merged
steipete merged 4 commits into
mainfrom
claude/test-suite-performance-8a87df
Jul 16, 2026
Merged

perf(test): fix env-dependent failures, stop verifier lazy-fetch, balance CI stripes#109332
steipete merged 4 commits into
mainfrom
claude/test-suite-performance-8a87df

Conversation

@steipete

Copy link
Copy Markdown
Contributor

What Problem This Solves

Test-suite performance work from a measured full-suite analysis (per-file JSON timing of all 12 full-* vitest shard configs, CI job timing from three recent runs, CPU profiles of the worst offenders):

  1. The single slowest test file in the repo, test/scripts/release-ci-summary.test.ts, took 306.9s locally: one test forges an unknown SHA and git show turned that into a hanging promisor network fetch on partial clones — inside a release-evidence security check.
  2. commands-name/commands-login each imported the full command-handler barrel (~1,700 modules) for one registration assertion (12.6s / 4.6s per file, ~98% import time).
  3. Five subagent command test files each re-evaluated the auto-reply module graph (the non-isolated runner resets modules per file).
  4. CI stripes for auto-reply-commands, agents-core-runner-cli, and core-tooling were split round-robin by filename, packing whale files together: auto-reply-reply-commands-1 ran ~220s while its siblings idled at 33s/24s.
  5. Three tests failed on real dev checkouts while CI stayed green (environment-dependent): canvas A2UI fixtures were shadowed by a locally built a2ui.bundle.js, speech-core prefs wrote through the macOS /tmp symlink which fs-safe rejects, and the bundled-plugin install test hardcoded extensions/discord while built checkouts resolve bundled sources to dist/extensions/discord.

Why This Change Was Made

Maintainer-requested (reduce CI wall time; fix any test failures hit along the way). CI wall time ≈ the longest vitest shard job; per-job setup is only ~25s, so slow files and unbalanced stripes are the lever.

User Impact

  • Contributors on partial clones or built checkouts get a green, fast suite: release-ci-summary 306.9s → 0.73s, canvas/speech-core/plugins-install tests now pass on macOS dev machines.
  • Release operators: evidence verification no longer depends on network lazy-fetch state; missing blobs fail closed as before (GIT_NO_LAZY_FETCH=1 on the blob read only).
  • CI: balanced stripes (greedy LPT over advisory per-file cost hints, deterministic) shorten the critical-path node-test jobs; command tests drop ~10s of redundant import work locally, more on 2-worker CI runners.

Evidence

  • release-ci-summary.test.ts: 41 tests, 306.9s (1 failing locally) → 0.73s, 41/41 after GIT_NO_LAZY_FETCH.
  • commands-name 12.6s → 1.4s, commands-login 4.6s → 1.0s, new commands-handlers.registration.test.ts 4.2s (barrel paid once); merged commands-subagents.test.ts 15 tests, 1.8s (one graph eval instead of five).
  • Canvas server.test.ts 17/18 → 18/18; speech-core tts.test.ts 43/44 → 44/44; commands-plugins.install.test.ts 27/28 → 28/28 (all previously CI-green but failing on built/partial-clone dev checkouts; root causes profiled, not guessed).
  • Striping: new distribution splits the whales (commands-status, cli-runner.spawn, cli-runner.reliability, pr-operation-lock, crabbox-wrapper, ...) across stripes; stripe group hints equalized at the old totals. test/scripts/ci-node-test-plan.test.ts 19/19, plus ci-changed-node-test-plan / ci-run-node-test-shard / ci-workflow-guards all green (109 passed, 1 skipped).
  • node scripts/check-deadcode-exports.mjs clean (new canvas test-api.ts seam follows the matrix test-api.ts precedent).
  • Codex autoreview (gpt-5.6-sol, xhigh): clean, no accepted/actionable findings (three runs across the work).

steipete added 4 commits July 16, 2026 13:27
…heckouts

Canvas A2UI fixtures now pin the resolved root through a new canvas
test-api hook: dev checkouts with a locally built a2ui.bundle.js won
candidate resolution and shadowed the fixture root. Speech-core prefs
tests resolve the canonical tmpdir (macOS /tmp symlink breaks fs-safe
store roots), and the bundled plugin install test asks discovery for
the canonical bundled path instead of hardcoding the source tree.
git show against a forged or missing SHA lazily fetched from the
promisor remote on partial clones, hanging a security check on network
state (266s observed). GIT_NO_LAZY_FETCH pins verification to local
objects; missing blobs fail closed as before.
…mmand tests

commands-name and commands-login imported the full handler barrel
(~1,700 modules) for registration assertions; those now live in a
dedicated registration test, dropping the two files from 12.6s/4.6s to
1.4s/1.0s locally. Five subagent command test files merged into one so
the auto-reply module graph evaluates once instead of five times.
Round-robin striping packed the whale files together, leaving
auto-reply-commands-1 at ~220s while sibling stripes idled at ~30s.
Greedy LPT over advisory per-file seconds hints balances the commands,
cli-runner, and tooling stripes deterministically; group hints for the
restriped bins are equalized at the old totals.
@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts size: XL maintainer Maintainer-authored PR labels Jul 16, 2026
@steipete
steipete merged commit 093b885 into main Jul 16, 2026
141 of 144 checks passed
@steipete
steipete deleted the claude/test-suite-performance-8a87df branch July 16, 2026 20:38
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 17, 2026
…ance CI stripes (openclaw#109332)

* test: fix environment-dependent failures on built and partial-clone checkouts

Canvas A2UI fixtures now pin the resolved root through a new canvas
test-api hook: dev checkouts with a locally built a2ui.bundle.js won
candidate resolution and shadowed the fixture root. Speech-core prefs
tests resolve the canonical tmpdir (macOS /tmp symlink breaks fs-safe
store roots), and the bundled plugin install test asks discovery for
the canonical bundled path instead of hardcoding the source tree.

* fix(release): keep evidence verifier blob reads local-deterministic

git show against a forged or missing SHA lazily fetched from the
promisor remote on partial clones, hanging a security check on network
state (266s observed). GIT_NO_LAZY_FETCH pins verification to local
objects; missing blobs fail closed as before.

* perf(test): pay the command-handler barrel once and merge subagent command tests

commands-name and commands-login imported the full handler barrel
(~1,700 modules) for registration assertions; those now live in a
dedicated registration test, dropping the two files from 12.6s/4.6s to
1.4s/1.0s locally. Five subagent command test files merged into one so
the auto-reply module graph evaluates once instead of five times.

* perf(ci): stripe node test bins by measured file cost

Round-robin striping packed the whale files together, leaving
auto-reply-commands-1 at ~220s while sibling stripes idled at ~30s.
Greedy LPT over advisory per-file seconds hints balances the commands,
cli-runner, and tooling stripes deterministically; group hints for the
restriped bins are equalized at the old totals.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer Maintainer-authored PR scripts Repository scripts size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant