Skip to content

test: stabilize load-sensitive test families that break gates under parallel load#100420

Merged
steipete merged 2 commits into
mainfrom
fix/stabilize-load-sensitive-tests
Jul 5, 2026
Merged

test: stabilize load-sensitive test families that break gates under parallel load#100420
steipete merged 2 commits into
mainfrom
fix/stabilize-load-sensitive-tests

Conversation

@steipete

@steipete steipete commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Six test families fail under parallel load while passing in serial isolation — they have repeatedly broken local landing gates and CI reruns (observed across five consecutive gate runs on a busy machine): the WhatsApp monitor-inbox media test, four tooling subprocess/signal suites (run-vitest wrapper cleanup, kitchen-sink fixture-server startup, write-cli-startup-metadata descendants, upgrade-survivor probe timing), and the Matrix crypto-bootstrap mock.

Why This Change Was Made

Each family's load sensitivity had a specific mechanism, fixed at the source rather than by inflating limits: subprocess fixtures now use a shared named LOAD_SENSITIVE_PROCESS_TIMEOUT_MS where the timeout only guards against genuinely broken fixtures (readiness/close signals do the real synchronization); the upgrade-survivor probe tests got injectable time so total-timeout ASSERTIONS stay meaningful instead of racing the wall clock; a child fixture pins scheduling with a setTimeout shim; the WhatsApp media-path test-support and Matrix mocks were made complete for everything production touches (per the repo's vi.mock completeness rule). No production code changed; no baselines/snapshots touched.

User Impact

None at runtime. Contributors and landing pipelines stop losing runs to these six families; test-suite results become trustworthy under machine load.

Evidence

  • Per-file focused runs all green; three-run stress loops on the two worst offenders: run-vitest 46/46 ×3, upgrade-survivor 11/11 ×3.
  • Matrix approval + crypto files: 30 passed.
  • Codex autoreview: one accepted finding during development (ES2024-only Promise.withResolvers replaced with ES2023-compatible resolver); final run clean.
  • Implementation by Codex (gpt-5.5) from a frozen spec; reviewed by the orchestrating session.

@openclaw-barnacle openclaw-barnacle Bot added channel: matrix Channel integration: matrix channel: whatsapp-web Channel integration: whatsapp-web size: M maintainer Maintainer-authored PR labels Jul 5, 2026
@clawsweeper

clawsweeper Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 5, 2026, 3:37 PM ET / 19:37 UTC.

Summary
This PR stabilizes load-sensitive Matrix, WhatsApp, and script test paths by replacing polling or wall-clock assumptions with pending-work signals, complete mocks, deterministic probe imports, and bounded process fixture waits.

PR surface: Source +20, Tests +37. Total +57 across 6 files.

Reproducibility: no. high-confidence independent reproduction was established in this read-only review. The PR body reports repeated parallel-load gate failures and focused stress loops, and source inspection confirms the touched tests exercise real pending-work, process-signal, and probe-timeout paths.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: migration/backfill/repair: test/scripts/upgrade-survivor-probe-gateway.test.ts, unknown-data-model-change: test/scripts/upgrade-survivor-probe-gateway.test.ts, unknown-data-model-change: test/scripts/write-cli-startup-metadata.test.ts, vector/embedding metadata: test/scripts/write-cli-startup-metadata.test.ts. Migration or upgrade compatibility proof is recorded; maintainers should verify it before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] The read-only review did not independently reproduce the reported five-run parallel-load failure; it relies on the PR body's stress-run summary, source-path plausibility, and exact-head CI status.

Maintainer options:

  1. Decide the mitigation before merge
    Keep this as the active test-stability PR and merge only after maintainer review and exact-head CI remain green; no cleanup close or replacement PR is supported.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P1] No repair lane is needed: the current head has no discrete review finding, and the remaining action is maintainer review plus normal merge gates for a protected test-only PR.

Security
Cleared: No security or supply-chain concern was found; the diff changes test and test-support files only and does not touch dependencies, workflows, secrets, package resolution, or production execution paths.

Review details

Best possible solution:

Keep this as the active test-stability PR and merge only after maintainer review and exact-head CI remain green; no cleanup close or replacement PR is supported.

Do we have a high-confidence way to reproduce the issue?

No high-confidence independent reproduction was established in this read-only review. The PR body reports repeated parallel-load gate failures and focused stress loops, and source inspection confirms the touched tests exercise real pending-work, process-signal, and probe-timeout paths.

Is this the best way to solve the issue?

Yes, this appears to be the best maintainable shape for the reported gate instability. The PR tightens test synchronization at existing owner boundaries instead of changing production runtime behavior or adding broad timeout inflation.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a320f775f0f3.

Label changes

Label justifications:

  • P2: The PR targets recurring test/gate instability with limited blast radius and no runtime user-facing emergency.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external proof gate is not applied here because the PR is maintainer-labeled and test-only; PR-body test-run notes and exact-head CI are supplemental review evidence.
Evidence reviewed

PR surface:

Source +20, Tests +37. Total +57 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 1 26 6 +20
Tests 5 135 98 +37
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 6 161 104 +57

What I checked:

  • Repository policy read and applied: Root policy plus scoped extension, test, and script guides were read; the test mock-completeness and wrapper/process-test guidance shaped the review. (AGENTS.md:1, a320f775f0f3)
  • Live PR state: Live GitHub reports the PR open, not draft, mergeable/CLEAN, with maintainer and P2 labels and six modified test/test-support files. (5f8300a80cdc)
  • Diff surface: The patch changes Matrix/WhatsApp test helpers and script tests only; it does not modify production runtime, dependencies, workflows, package metadata, config, or lockfiles. (5f8300a80cdc)
  • WhatsApp pending-work contract: Current main publishes pending-work count from handler, debounce, and flush state, supporting the PR's move from polling to monitor-owned drain signaling. (extensions/whatsapp/src/inbound/monitor.ts:420, a320f775f0f3)
  • WhatsApp test synchronization: The PR arms an onPendingWorkChanged callback before emitting the upsert and waits until observed pending work drains instead of using the old message-call polling plus setImmediate settling helper. (extensions/whatsapp/src/monitor-inbox.captures-media-path-image-messages.test-support.ts:60, 5f8300a80cdc)
  • Upgrade probe timing contract: Current main computes probe timeout from Date.now(), attempt deadlines, setTimeout, fetch, and bounded response reads, matching the PR's deterministic import-based test seam. (scripts/e2e/lib/upgrade-survivor/probe-gateway.mjs:108, a320f775f0f3)

Likely related people:

  • steipete: Path history shows Peter Steinberger as the most recent current-main contributor across the affected test and channel files, and the PR also comes from this same test-stability area. (role: recent area contributor; confidence: high; commits: f8016c44a4ba, 1d7d268a636b, 9f7aaa8ad764; files: test/scripts/upgrade-survivor-probe-gateway.test.ts, test/scripts/run-vitest.test.ts, extensions/whatsapp/src/inbound/monitor.ts)
  • vincentkoc: History shows repeated earlier work on Vitest process cleanup, no-output watchdogs, and release/test stabilization around the script surfaces touched by this PR. (role: feature-history owner; confidence: high; commits: e085fa1a3ffd, 329bba6cca58, 9dbbe740a4d4; files: scripts/run-vitest.mjs, test/scripts/run-vitest.test.ts, test/scripts/upgrade-survivor-probe-gateway.test.ts)
  • mcaxtr: WhatsApp path history includes substantial inbound state and lifecycle work by Marcus Castro near the monitor pending-work surface used by the PR test helper. (role: adjacent owner; confidence: medium; commits: 458a52610a4d, aa023e428306, da1da6110209; files: extensions/whatsapp/src/inbound/monitor.ts)
  • gumadeiras: Matrix approval reaction history points to the original approval-reaction shortcut work that the modified Matrix approval-reactions test exercises. (role: feature introducer; confidence: medium; commits: 0aaf7531481d; files: extensions/matrix/src/approval-reactions.ts, extensions/matrix/src/approval-reactions.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-07-05T18:52:40.634Z sha 272c166 :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 5, 2026
@steipete
steipete force-pushed the fix/stabilize-load-sensitive-tests branch from 272c166 to 5f8300a Compare July 5, 2026 19:23
@steipete
steipete force-pushed the fix/stabilize-load-sensitive-tests branch from 5f8300a to 294c4d3 Compare July 5, 2026 19:39
@steipete
steipete merged commit a04b6ce into main Jul 5, 2026
90 checks passed
@steipete
steipete deleted the fix/stabilize-load-sensitive-tests branch July 5, 2026 19:53
@steipete

steipete commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 6, 2026
…arallel load (openclaw#100420)

* test: stabilize process-heavy fixtures under load

* test: stabilize channel mocks under load
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: matrix Channel integration: matrix channel: whatsapp-web Channel integration: whatsapp-web maintainer Maintainer-authored PR P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant