Skip to content

fix(qa): stop suite execution after the first failed scenario#113843

Merged
steipete merged 1 commit into
mainfrom
codex/fix-autoqa-qa-suite-fail-fast-e279
Jul 25, 2026
Merged

fix(qa): stop suite execution after the first failed scenario#113843
steipete merged 1 commit into
mainfrom
codex/fix-autoqa-qa-suite-fail-fast-e279

Conversation

@steipete

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes an issue where QA operators running fail-fast suites could not use the flag through the normal host or profile CLI, and mixed flow, native, script, and Multipass runs continued launching work after the first failure. Missing started scenario results could also produce misleading successful or incomplete evidence.

Why This Change Was Made

Routes fail-fast through the canonical host, profile, Multipass, native runner, and existing stop-aware unified scheduler. Separately tracks submitted, started, and actually returned scenarios so execution, reports, and evidence all stop at the same failure boundary. Synthesizes canonical failed evidence for missing started results, retains producer-owned evidence, and preserves ordinary weighted parallel scheduling when fail-fast is disabled.

User Impact

QA users can stop unnecessary gateways, browser tests, scripts, and VM work immediately after the first genuine failure. The generated suite report, summary, and evidence accurately distinguish failed started scenarios from work that never ran.

Evidence

  • Regression-first proof reproduces the rejected host/profile flag, missing Multipass guest argument, mixed-partition and native-runner continuation, missing-result false successes, grouped-native execution, credential-failure tails, and evidence-artifact mismatches.
  • All 11 QA, Matrix transport, Multipass, scheduler, catalog, CLI, and evidence sibling suites pass: 361 tests.
  • Real source-checkout OpenClaw CLI starts a private QA lab, an actual mock provider, and a real isolated Gateway: passing direct-message scenario.
  • Real source-checkout CLI starts a separate isolated Gateway against live OpenAI openai/gpt-5.4: passing direct-message scenario.
  • Complete remote pnpm check:changed passes both production and test typechecking, formatting, plugin API and ownership guards, all database/runtime guards, import-cycle checks, and a 7,722-file lint scan with zero warnings or errors.
  • Fresh independent structured review: patch is correct, confidence 0.91, zero actionable findings.
  • Author and committer verified as Peter Steinberger [email protected].

@clawsweeper

clawsweeper Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(qa): stop suite execution after the first failed scenario This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@steipete
steipete merged commit d8f099e into main Jul 25, 2026
138 of 140 checks passed
@steipete
steipete deleted the codex/fix-autoqa-qa-suite-fail-fast-e279 branch July 25, 2026 20:26
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant