Skip to content

test: wire Control UI suite into CI, fix its broken tests, drop dead harness surface#104361

Merged
steipete merged 4 commits into
mainfrom
claude/test-harness-streamline
Jul 11, 2026
Merged

test: wire Control UI suite into CI, fix its broken tests, drop dead harness surface#104361
steipete merged 4 commits into
mainfrom
claude/test-harness-streamline

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Closes #104321

What Problem This Solves

Fixes an issue where the Control UI Vitest suite (3283 tests) was red on main with nobody noticing, because no CI workflow runs it: 7 tests across ui/src/pages/chat/chat-realtime.test.ts and ui/src/pages/chat/components/chat-pull-requests.test.ts failed in any full-suite run. Also resolves the remaining test-harness integrity items from the follow-up audit: an unreachable extension routing entry, ~35 lines of never-populated channel-override machinery, unguarded hardcoded path maps in the Vitest wrapper, dead coverage thresholds, and a Node 22 lower-bound compat job that only ran on manual dispatch.

Why This Change Was Made

The broken ui tests were diagnosed as stale tests, not product regressions: the ui Vitest config runs isolate: false, and these tests (added after that flip) relied on vi.mock module factories that only take effect when their file loads first in a shared worker — green solo, red in the full suite. They now use RealtimeTalkSession prototype spies and real jsdom storage. To keep this class of rot from recurring, a new gated checks-ui CI job runs the suite whenever ui-affecting paths change (runUiTests changed-scope output), measured at ~13s for the suite plus ~13s Chromium provisioning on a Testbox-class runner — negligible CI cost, and zero for non-ui PRs. The dead-surface deletions follow the repo lean-code policy; the coverage thresholds were unenforced policy surface (docs updated accordingly); the weekly node22-compat workflow exercises the supported lower Node bound with the same command set as the existing dispatch-only ci.yml job.

User Impact

None at product runtime. Maintainers get: ui regressions caught on PRs, a weekly Node 22 compatibility signal, fewer dead harness paths, and honest test:coverage documentation.

Evidence

  • Remote (Blacksmith Testbox through Crabbox, leases tbx_01kx8747whetcg3jdej2s266rb / tbx_01kx86pdwv64chz5pjsjbnqyth): full Control UI suite green (214 files / 3283 tests, ~13s), focused harness meta-tests green (391 tests), pnpm test:changed green, pnpm check:changed (typecheck core tests, lint core, lint scripts, guards) green, pnpm check:workflows green — final combined rerun exit 0.
  • The rewritten ui tests pass in both orders (solo and full suite). A local Node 26.5 jsdom quirk makes window.localStorage undefined regardless of this diff; CI truth is Linux Node 24, where everything is green.
  • Structured autoreview (codex, gpt-5.6-sol, thinking xhigh) on the final branch diff: clean, "patch is correct (0.93)", no actionable findings.
  • Item deliberately deferred: routing test/** TS edits to a typecheck lane. test/tsconfig/tsconfig.core.test.json does not include root test/**/*.ts (the test: fix changed-run routing, dedupe gateway package lanes, repair force-rerun triggers #104287 envDir error was caught only transitively), so honest coverage needs a dedicated root-test tsgo lane — same shape as the in-flight tsgo:scripts lane work. Follow-up noted on test harness follow-up: ui suite red with no CI lane, changed-gate typecheck gap, dead routing machinery, unenforced coverage thresholds #104321.
  • Note for reviewers: an in-flight branch adds a tsgo:scripts lane touching changed-lanes.mjs/check-changed.mjs/ci.yml; whichever lands second rebases (hunks are disjoint).

steipete added 3 commits July 11, 2026 10:38
…harness surface

- fix 7 Control UI tests broken on main: module-mock factories are unreliable
  under isolate:false shared workers; use RealtimeTalkSession prototype spies
  and real jsdom storage instead (chat-realtime, chat-pull-requests)
- add gated checks-ui CI job (runUiTests changed-scope output): chromium
  provisioning, lint:ui:no-raw-window-open, pnpm --dir ui test; ~30s on a
  4vcpu runner, runs only when ui-affecting paths change
- add weekly node22-compat workflow exercising the supported lower Node bound
  with the same command set as the dispatch-only ci.yml job
- delete the unreachable channels entry in EXTENSION_TEST_CONFIG_ROUTES and
  the never-populated extensionRoutedChannelTestFiles override machinery
  (born empty in 2ccb5cf); resolved globs are provably identical
- guard run-vitest.mjs hardcoded path maps with existence tests so renames
  cannot silently drop extended stall watchdogs
- remove unenforced coverage thresholds; test:coverage is informational, docs
  updated (reference/test, help/testing, plugins/sdk-testing)

Closes #104321
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation app: web-ui App: web-ui scripts Repository scripts size: M maintainer Maintainer-authored PR labels Jul 11, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 413316e2af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml
run: pnpm lint:ui:no-raw-window-open

- name: Test Control UI
run: pnpm --dir ui test

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Run the full Control UI test script

For UI source changes outside ui/src/i18n, preflight sets run_ui_tests but not run_control_ui_i18n, so this new job runs only the raw ui Vitest suite plus the window-open lint and skips the existing pnpm test:ui script’s pnpm ui:i18n:check step. That means a normal UI component change can introduce hard-coded copy or stale locale output without this CI lane catching it; use pnpm test:ui here or add the i18n check to the same checks-ui job.

Useful? React with 👍 / 👎.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 11, 2026
@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed July 11, 2026, 5:53 AM ET / 09:53 UTC.

Summary
The PR adds a path-gated Control UI CI job, rewrites order-sensitive UI tests, adds a weekly Node 22 workflow, removes dead Vitest routing and coverage policy, and updates test documentation.

PR surface: Source +4, Tests +15, Docs 0, Config +67, Other +10. Total +96 across 19 files.

Reproducibility: yes. The changed-scope rules and workflow command establish a deterministic source-level path where an ordinary ui/** edit runs checks-ui without the i18n/raw-copy drift check.

Review metrics: 1 noteworthy metric.

  • New recurring automation: 1 scheduled workflow added. The weekly Node 22 lane establishes a recurring lower-bound build-and-smoke contract that maintainers must keep aligned with supported commands.

Stored data model
Persistent data-model change detected: serialized state: ui/src/pages/chat/components/chat-pull-requests.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #104321
Summary: This PR is the explicit candidate fix for the linked open follow-up audit issue; the earlier merged audit PR covers adjacent but distinct work.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Switch checks-ui to pnpm test:ui and rerun the focused workflow and changed-scope checks.

Risk before merge

  • [P1] Merging as written can yield false-green Control UI CI when ordinary pages or components introduce hard-coded copy or stale generated locale state.

Maintainer options:

  1. Use the canonical UI gate (recommended)
    Replace the duplicated Chromium, lint, and direct package-test steps with pnpm test:ui so every established Control UI invariant runs together.
  2. Add the omitted drift check
    Keep the split workflow steps but add pnpm ui:i18n:check, accepting ongoing duplication between CI and the canonical aggregate script.

Next step before merge

  • The protected maintainer label requires explicit maintainer handling; the concrete next action is to update the branch's UI job command and rerun its focused proof.

Security
Cleared: No concrete security or supply-chain regression was found in workflow permissions, action pins, dependencies, secret access, or execution paths.

Review findings

  • [P1] Run the canonical Control UI test gate — .github/workflows/ci.yml:914
Review details

Best possible solution:

Have checks-ui invoke the canonical pnpm test:ui aggregate, preserving one owner for locale drift, UI lint, Chromium setup, and the UI suite while retaining the remaining focused harness cleanup and Node 22 coverage.

Do we have a high-confidence way to reproduce the issue?

Yes. The changed-scope rules and workflow command establish a deterministic source-level path where an ordinary ui/** edit runs checks-ui without the i18n/raw-copy drift check.

Is this the best way to solve the issue?

No. Reusing the existing aggregate script is narrower and prevents the CI implementation from drifting away from the supported Control UI test contract.

Full review comments:

  • [P1] Run the canonical Control UI test gate — .github/workflows/ci.yml:914
    Use pnpm test:ui here, or add its i18n check. run_ui_tests covers ordinary ui/** edits, while run_control_ui_i18n does not, and the i18n checker scans pages and components for new raw copy. This lane can therefore pass while skipping an invariant already owned by the supported UI test script.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.97

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 14a056ac83a3.

Label changes

Label changes:

  • add P2: This is bounded CI and test-harness correctness work without a product-runtime outage.
  • add merge-risk: 🚨 automation: The new UI job can report success while skipping an invariant already owned by the canonical Control UI test command.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides after-fix Testbox live output for the full Control UI suite, focused harness tests, changed tests, changed checks, and workflow validation.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body provides after-fix Testbox live output for the full Control UI suite, focused harness tests, changed tests, changed checks, and workflow validation.

Label justifications:

  • P2: This is bounded CI and test-harness correctness work without a product-runtime outage.
  • merge-risk: 🚨 automation: The new UI job can report success while skipping an invariant already owned by the canonical Control UI test command.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body provides after-fix Testbox live output for the full Control UI suite, focused harness tests, changed tests, changed checks, and workflow validation.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides after-fix Testbox live output for the full Control UI suite, focused harness tests, changed tests, changed checks, and workflow validation.
Evidence reviewed

PR surface:

Source +4, Tests +15, Docs 0, Config +67, Other +10. Total +96 across 19 files.

View PR surface stats
Area Files Added Removed Net
Source 1 4 0 +4
Tests 9 124 109 +15
Docs 3 3 3 0
Config 2 67 0 +67
Generated 0 0 0 0
Other 4 15 5 +10
Total 19 213 117 +96

What I checked:

  • Partial UI gate introduced: The new checks-ui job provisions Chromium, runs the window-open lint, and executes the UI package tests directly, but does not run the i18n/raw-copy check included by the canonical aggregate command. (.github/workflows/ci.yml:914, 70c0ff801f24)
  • Path routing exposes the gap: The branch routes every ui/** edit to run_ui_tests, while run_control_ui_i18n is limited to i18n-owned paths and the i18n automation files. (scripts/ci-changed-scope.mjs:57, 70c0ff801f24)
  • Canonical aggregate command: Current main's supported test:ui command runs ui:i18n:check, the window-open lint, Chromium provisioning, and the UI suite as one curated entry point. (package.json:1944, 14a056ac83a3)
  • The omitted check covers ordinary UI files: The i18n checker scans Control UI app, component, library, and page sources for raw-copy drift, not only files beneath the i18n directory. (scripts/control-ui-i18n.ts:114, 14a056ac83a3)
  • Existing review identified the same defect: The current-head review thread calls out that ordinary UI changes trigger the new job without triggering the separate i18n lane and asks the branch to use pnpm test:ui or add the missing check. (.github/workflows/ci.yml:914, 70c0ff801f24)
  • Strong real-run evidence: The PR body reports after-fix Blacksmith Testbox runs covering the full Control UI suite, focused harness tests, changed tests, changed checks, and workflow validation with a final successful combined run. (70c0ff801f24)

Likely related people:

  • steipete: Recent merged work changed CI lane routing and related test-tooling surfaces, making this the strongest route for the focused workflow repair. (role: recent CI and tooling contributor; confidence: high; commits: 2e2366b6d394; files: .github/workflows/ci.yml, scripts/ci-changed-scope.mjs, package.json)
  • Vincent Koc: Current history ties the raw-copy drift scanner and aggregate UI test command area to this contributor, making them relevant to the invariant the new job currently omits. (role: Control UI i18n and test-surface contributor; confidence: medium; commits: cce3fd55c486; files: scripts/control-ui-i18n.ts, package.json, ui/AGENTS.md)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete
steipete merged commit 6268eee into main Jul 11, 2026
139 of 143 checks passed
@steipete
steipete deleted the claude/test-harness-streamline branch July 11, 2026 10:04
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

vincentkoc added a commit that referenced this pull request Jul 11, 2026
* origin/main:
  fix(cron): abort superseded reconciliation hooks (#104368)
  ci(release): expose Telegram runtime preflight stage (#104387)
  fix(status): avoid false shell-wrapper audit warnings (#81778)
  fix: retry live Gateway readiness proof (#104374)
  fix(exec-approval): stop misattributing Allow Always unavailability to policy (#97740)
  fix: preserve Mac bundle during live builds (#104376)
  chore(docs): translate with GPT-5.6 xhigh
  test: wire Control UI suite into CI, fix its broken tests, drop dead harness surface (#104361)
  fix(ui): hide group submenu separator when New group is the only entry (#104370)
  fix(discord): reset progress drafts across queued turns (#102341)
  docs(cli): clarify that exec-policy show and approvals get exclude per-session /exec overrides (#94999)
  fix(discord): single-source thread-binding default placement and guard artifact parity (#104342)
  fix(agents): add tool-activity heartbeat to keep subagent alive during tool calls (#95536)
  fix(maint): reuse recent same-PR hosted gates (#104355)
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 12, 2026
…harness surface (openclaw#104361)

* test: wire Control UI suite into CI, fix its broken tests, drop dead harness surface

- fix 7 Control UI tests broken on main: module-mock factories are unreliable
  under isolate:false shared workers; use RealtimeTalkSession prototype spies
  and real jsdom storage instead (chat-realtime, chat-pull-requests)
- add gated checks-ui CI job (runUiTests changed-scope output): chromium
  provisioning, lint:ui:no-raw-window-open, pnpm --dir ui test; ~30s on a
  4vcpu runner, runs only when ui-affecting paths change
- add weekly node22-compat workflow exercising the supported lower Node bound
  with the same command set as the dispatch-only ci.yml job
- delete the unreachable channels entry in EXTENSION_TEST_CONFIG_ROUTES and
  the never-populated extensionRoutedChannelTestFiles override machinery
  (born empty in b479115); resolved globs are provably identical
- guard run-vitest.mjs hardcoded path maps with existence tests so renames
  cannot silently drop extended stall watchdogs
- remove unenforced coverage thresholds; test:coverage is informational, docs
  updated (reference/test, help/testing, plugins/sdk-testing)

Closes openclaw#104321

* test: pin OPENCLAW_CI_RUN_UI_TESTS in the manifest env expectation

* test(ui): capture the realtime start spy instead of referencing the unbound prototype method

* test(ui): raise ui vitest timeouts for real-browser layout tests on small CI runners
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 12, 2026
* origin/main:
  fix(cron): abort superseded reconciliation hooks (openclaw#104368)
  ci(release): expose Telegram runtime preflight stage (openclaw#104387)
  fix(status): avoid false shell-wrapper audit warnings (openclaw#81778)
  fix: retry live Gateway readiness proof (openclaw#104374)
  fix(exec-approval): stop misattributing Allow Always unavailability to policy (openclaw#97740)
  fix: preserve Mac bundle during live builds (openclaw#104376)
  chore(docs): translate with GPT-5.6 xhigh
  test: wire Control UI suite into CI, fix its broken tests, drop dead harness surface (openclaw#104361)
  fix(ui): hide group submenu separator when New group is the only entry (openclaw#104370)
  fix(discord): reset progress drafts across queued turns (openclaw#102341)
  docs(cli): clarify that exec-policy show and approvals get exclude per-session /exec overrides (openclaw#94999)
  fix(discord): single-source thread-binding default placement and guard artifact parity (openclaw#104342)
  fix(agents): add tool-activity heartbeat to keep subagent alive during tool calls (openclaw#95536)
  fix(maint): reuse recent same-PR hosted gates (openclaw#104355)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app: web-ui App: web-ui docs Improvements or additions to documentation maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. scripts Repository scripts size: M status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test harness follow-up: ui suite red with no CI lane, changed-gate typecheck gap, dead routing machinery, unenforced coverage thresholds

1 participant