Repository navigation
fix(tests): prevent onboarding tests from crashing on macOS - #2883
johnmatthewtennant wants to merge 5 commits into
Conversation
Co-authored-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <[email protected]>
Co-authored-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <[email protected]>
Co-authored-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <[email protected]>
Co-authored-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <[email protected]>
Co-authored-by: Zack <04f47b290b155b3b8b782b11501cdb15f1e21e82dfceefd01ff0ed405459d927@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <[email protected]>
cba388d to
94a0e7f
Compare
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 LGTM. The not(test) gate keeps the AppKit call in the app build and drops it only from the lib unit-test build, where MockRuntime has no real NSWindow. The macOS CI step now runs both filters (61 tests, including onboarding_snapshot_is_kept_until_next_launch_restore). The publication fixture's PATH and environment are now fully explicit, and the script only needs the fake gh/sleep plus grep.
Nit, nonblocking: since env no longer spreads process.env, BASH_ENV can't reach the shell, and a non-interactive bash -c never reads profile or rc files. That makes --noprofile --norc -p redundant, so /bin/bash -eo pipefail -c would be enough.
wesbillman
left a comment
There was a problem hiding this comment.
🤖 Automated source review by Star Lord via Wes’s account.
No actionable findings at 94a0e7fab4a924806044e300d5bd494ecdba6872 against target/merge-base ab344439a38248bb7088adf247167827e5fe8f4b. Reviewed the unit-test-only AppKit exclusion, combined macOS Cargo selection, and publication fixture command/environment isolation with the retained retry and release-preservation assertions. The native animation call remains enabled in ordinary macOS builds.
Validation: source-only review; no tests or app code executed locally. The hosted CI run reports success at this head; its macOS artifact records 61 passing selected library tests, including the onboarding snapshot regression (2m13s step wall time, 3.71s test execution). This does not establish native GUI behavior or a before/after performance improvement. The author’s base-crash and environment-injection probes were not independently reproduced; Windows intentionally skips the POSIX publication fixture.
…in the measurements lane (#3447) ## Why `browser-host/relay-broker-live.test.mjs`, "real broker keeps 100,000 joined channels and delivers outside the first batch", times out intermittently in the shared JavaScript shard. | Sighting | Lane | |---|---| | [7 Nov, PR, passed on re-run](https://github.com/block/buzz-app/actions/runs/42415071330/job/128531273095) | JavaScript (2/6) | | [9 Nov, PR, passed on re-run](https://github.com/block/buzz-app/actions/runs/42600556170/job/129103660721) | JavaScript (2/6) | **This PR edits `.github/workflows/ci.yml`.** It adds one step to the existing Browser measurements job. The required gate is unchanged: no job was added or removed, and the measurements result that `CI required` already checks now carries the scale check. ## Root cause The test asserts wall-clock time: about 20 s of single-threaded CPU has to finish inside a 30 s `vi.waitFor` budget. Vitest hashes files to shards, so the test always shared shard 2/6 with `InboxPage.test.tsx` and `vite-config.test.mjs`, under two workers. - `main` passed at 22.5 s. - The two failing jobs reached the budget at 33.9 s and 32.8 s, with the broker connected and no channels reported joined yet. - Their whole shard ran 440 to 446 s against 366 s on `main`. - Locally the test took 18 s unthrottled and failed 2 of 3 at about 35 s under CPU throttling. The 30 s budget is a documented performance contract, so it is not raised. ## Change The test body moves into a helper shared by two tests in the same file. - **"real broker delivers outside the first joined batch"** joins one channel more than a live batch and runs in the shared shards. It keeps the behavioural contract: one socket, the first live-only REQ names a full batch that excludes the last channel, an event in that channel is delivered, and swapping it for a new channel establishes the new one and retires the old route. The lone overflow channel replays inside its own REQ, so this case expects one live-only batch REQ, not two. - **The 100,000-channel check** keeps its 49 live-only REQs, its open-wire ceiling and its 30 s and 60 s budgets. It is gated with `test.skipIf(!process.env.BUZZ_BROKER_SCALE)`, the same pattern the Claude Code protection live test uses. The gated check runs as a step of the Browser measurements job, which already runs serially on an isolated runner. The step sets `BUZZ_BROKER_SCALE`, filters by title through `ci-test-report.mjs`, and uploads its report with the job's evidence artifact. A title filter that matches nothing is a green run of zero tests, so the step then requires exactly one passed test. `docs/contributing.md` lists the step under the lane. No budget, assertion or failure path was weakened, and no engine was skipped. ## Evidence Measured locally on an idle 32-core Linux host, Vitest 5.0.4. | Measurement | Before | After | |---|---|---| | Scale check alone | 14.8 s test time (`vitest run -t "100,000"`, 15.3 s total) | 15.0 s test time, 16.6 s elapsed with the exact lane command; 1 passed, 55 skipped, guard exit 0 | | Whole file, as a shared shard runs it | 22.2 s, 55 tests | 7.5 s, 55 passed and 1 skipped | | New behavioural case | | 10 of 10 fresh runs passed, 231 to 277 ms | | Guard with the gate unset, `-t "100,000"` | | vitest exits 0 with 56 skipped; the guard exits 1 | Hosted, before, from a green `main` run: Browser measurements 10 m 34 s, JavaScript (2/6) 7 m 31 s. Biome is clean on the test file and the workflow parses. The pre-push type check and the file's related unit tests passed on this branch's head. `tests/integration/browser-ci.test.mjs` pins the upload paths of the **Measurement evidence** step, in the case "classic-scrollbar cases run exactly once, after the measurements, without clearing their evidence". The second commit adds `test-results/broker-scale` to that expected list. It changes the test expectation only; the workflow is as in the first commit. | Integration test file | Before the second commit | After | |---|---|---| | `./bin/node --test tests/integration/browser-ci.test.mjs`, macOS, Node v24.18.0 | 6 passed, 1 failed on the upload-path assertion | 7 passed, 0 failed | That assertion is the only pin of the measurements job's steps or paths. The lane's other integration test files and the Linux container the lane uses in CI were not run locally. ## Not confirmed, not run - Hosted timings after the change are not known yet. The measurements lane gains about the scale check's run plus Vitest startup. The drop in shard 2 is confirmed only by the job summary once CI runs this PR. Reviewers should compare both there. - The local timings are from Linux and are not hosted CI numbers. - No human has tested this yet, so `buzz-review-completed` is not set. ## Left alone - `InboxPage.test.tsx` shares the same slow shard with no race found. It needs fixture-cost profiling rather than a timeout change. - Open PRs #3050, #3046, #2883, #2836, #2014, #1140, #1105 and #873 also edit `ci.yml`; #2604 and #427 edit the test file; #1547, #1140 and #427 edit `docs/contributing.md`. A trial merge of each against this change is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <[email protected]> Signed-off-by: Matt Toohey <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
Why
The existing onboarding snapshot regression segfaults on macOS. Tauri's MockRuntime returns a placeholder NSView pointer, and
onboarding_preserve_windowsends it an AppKit message before checking the saved snapshot.The release-publication fixture also inherits host executables and GitHub authentication, so a missed fake
ghcan reach the real CLI.What
Exclude the native animation call from unit-test builds, preserving it in the app. Run the existing protection and window-state tests together in one macOS Cargo invocation, sharing the Rust test build. Isolate the publication fixture’s executable path and environment, disable shell startup processing, and include stderr when its expected exit status fails.
Validation
On macOS,
./bin/cargo test --manifest-path src-tauri/Cargo.toml --locked --lib onboarding_snapshot_is_kept_until_next_launch_restoreexits with SIGSEGV on the base revision and passes with this fix. The retained regression calls the onboarding command with a mock window and verifies that the snapshot survives until the next launch restore.With an inert external
ghon the parent PATH, removing or disabling the fixture’s fake reaches that external executable before isolation. After isolation, both cases stop with command-not-found or permission-denied diagnostics; an intact fake receives no injected parent GitHub tokens. The original broad-run failure’s trigger remains unconfirmed.