This repository was archived by the owner on Nov 7, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 7, 2026. It is now read-only.
test suite has at least 5 environment-sensitive tests: which one fails varies by run and machine #10
Closed as not planned
Description
What
Now that CI runs on this fork (#7), the suite is visibly unstable. Across four runs of the same code it produced four different failure sets.
| run | environment | failed |
|---|---|---|
| local | this box | tool.write > file permissions > sets file permissions when writing sensitive data |
| local | this box | tool execution produces non-empty session diff (snapshot race) |
| CI #7 run 1 | ubuntu-latest | opencode run (non-interactive subprocess) > exits nonzero promptly when the model is unknown (regression for #27371) |
| CI #7 run 2 | ubuntu-latest | unit (linux) passed - nothing failed |
| CI #7 run 2 | ubuntu-latest | e2e: regression/session-timeline-transport.spec.ts › reconnects after a stream error, plus a historical-diff-summary-overflow spec |
The instability itself is the finding. No single test is reliably broken, and tool.write - which fails consistently on this box - passes in CI.
Evidence each is environment-sensitive, not a regression
unit (linux)failed then passed on identical commits (ci: point test and typecheck at this fork, on GitHub-hosted linux runners #7 changes only two YAML files, no code).snapshot-tool-racefails in a full-suite run and passes in isolation - classic contention.opencode run ... exits nonzero promptlytook 15.7 s before failing, on a test whose subject is promptness. That reads like a timeout under load rather than a behavioural break.- The e2e failures are Playwright
toBeVisibletimeouts at 10 s (Error: element(s) not found), which are timing assertions against a live UI.
Why it matters
A suite where the red job changes every run cannot gate anything. The immediate cost is that every PR author has to re-derive whether their failure is real - which happened twice while landing #6 and #7, once needing a WIP-commit-and-revert against origin/swxtch to prove a failure was pre-existing.
Suggested direction
- Quarantine or fix the five named above; they are concrete and reproducible enough to start.
- For the timeout-shaped ones (
exits nonzero promptly, the Playwright specs), raise or make the deadline load-relative rather than a fixed wall-clock, since GitHub-hosted runners are slower than the Blacksmith images these thresholds were presumably tuned against. snapshot-tool-racepasses in isolation, so it needs isolation or a real synchronisation point, not a longer timeout.
Related: #7 (CI enablement), #6 (the first feature PR to get checks).
Activity
brentyates-swx commentedon Sep 22, 2026
New evidence, 2026-09-22 — three more unit tests, and a re-verification note
Observed while landing #9 (PR #22), on a branch whose diff touches nothing under
packages/— only.github/workflows,script/and.gitignore. Both of that PR's new CI steps passed in the same run, so a causal link from the change is implausible.Three more environment-sensitive unit tests, same family as the one already listed
Run 35719872063,
unit (linux):--format json records an unknown stream finish and continuation--format json preserves reasoning, tool, and continuation orderingunknown stream finish preserves partial output and continuesAll three in
opencode run (non-interactive subprocess)—packages/opencode/test/cli/run/run-process.test.ts— and all three within 90 ms of 30 seconds, which is thetimeoutMs: 30_000deadline that file uses. That is the signature of a load-dependent deadline, not three separate behavioural breaks.One e2e failure, which retried three times
Same run,
e2e (linux): a single spec failed,e2e/regression/session-todo-*("...ying it across session tabs"), failing on the initial attempt and both retries. Worth separating from the timeout-shaped ones: something that fails three times consecutively is more reproducible than the PlaywrighttoBeVisibletimeouts this issue already describes, and may be a real defect rather than a flake.Correction to this issue's own list
The test this issue names as
exits nonzero promptly when the model is unknown (regression for #27371)no longer exists under that name. In the current tree it isexits nonzero without hanging when the model is unknown, andpackages/opencode/test/cli/run/run-process.test.ts:75now carries:So that one has already had this issue's own prescription #2 applied to it — the deadline was demoted to a backstop and the assertion moved to the exit code. It should be struck from the list of five, and the fix that was applied there is the template for the three above.
Implication for the suggested direction
Item 2 of this issue ("raise or make the deadline load-relative") should be narrowed:
run-process.test.tsalready shows the better pattern, which is not a longer deadline but asserting the observable outcome and keeping time only as a bounded backstop.expect(result.durationMs).toBeLessThan(30_000)at line 339 of that same file is an example of the old pattern still in place.Re-verification note
I could not establish these as pre-existing from CI history: the base branch's last two runs (
bbe8578ec,148fdcdfd) were both green, including on the exact commit this branch is based on. Combined with the diff touching no product code, the reading is intermittent rather than a regression — but that is inference, not proof, because every subsequent run on the branch was cancelled by the next push before completing. Whoever picks this up should run the three named tests in isolation and underbun turbo teston a clean checkout ofswxtchbefore assuming either way.brentyates-swx commentedon Sep 22, 2026
Confirmed intermittent, with a clean three-run sequence
Follow-up to my note above, which said the reading was inference rather than proof. It is now proof. Three completed runs of the same branch, whose diff touches nothing under
packages/:961448dcbbae94a2d32473ff53cNo product code changed between them — the diffs are
script/and.github/workflowsonly. The failing set shrank 3 → 1 → 0 while the e2e job flipped fail → pass → pass. That is the signature this issue describes, now with a monotonic sequence rather than a single observation.The three unit tests, for the record, all in
packages/opencode/test/cli/run/run-process.test.ts:--format json records an unknown stream finish and continuation--format json preserves reasoning, tool, and continuation orderingunknown stream finish preserves partial output and continuesAll three land within 90 ms of the 30-second
timeoutMsthat file uses, which is why they are a load story and not three behavioural breaks. The first of the three failed in both failing runs, so if anything is worth fixing first it is that one.The e2e spec that failed on the first run —
e2e/regression/session-todo-*— failed on its initial attempt and both retries, then passed entirely on the next two runs. Worth keeping distinct from the timeout-shaped ones when someone works this: three consecutive failures followed by clean passes is a different shape from a 10-secondtoBeVisibletimeout.Practical note for whoever picks this up: every intermediate run on that branch was cancelled by the next push (
cancel-in-progress: true), so gathering this took holding off on pushing. If you are trying to characterise a flake here, stop pushing first — otherwise you getcancelledrather than data.brentyates-swx commentedon Sep 22, 2026
Fourth data point: 3 → 1 → 0 → 2
Continuing the sequence from my comment above, same branch, still no change to anything under
packages/:961448dcbbae94a2d32473ff53ca95c2dfd3The count is not monotonic — it went to zero and came back — which rules out the last remaining alternative reading, that something in the branch was progressively fixing it. Four runs, four different failure counts, one file, every failure within ~400 ms of the 30-second deadline.
The two that failed most recently:
unknown stream finish preserves partial output and continues--format json records an unknown stream finish and continuationThe second has now failed in three of the four runs, making it the most load-sensitive of the group and the obvious first candidate for the
run-process.test.ts:75treatment (assert the observable outcome; keep the deadline as a bounded backstop only).Also worth recording for whoever picks this up: e2e has now passed three runs in a row after its single failure, and that failure took three attempts to reproduce within one run (initial + 2 retries) before passing cleanly afterwards. Whatever is wrong there behaves differently from the unit timeouts and should not be lumped in with them.
test: make a deadline kill say so, and stop asserting this machine's …
test(e2e): fix the duplicate sessionStatus key in the todo-dock spec
ci: gate on the lint rules where a hit is almost certainly a bug
style: format the lint gate config
test(e2e): sample the todo dock until a condition, not for a fixed wi…
test(e2e): scale the shared app-readiness and expect timeouts
test: make the environment-sensitive tests attribute their own failur…
fix(e2e): repair the import my own sweep broke in scroll-interaction
test(e2e): scale the per-call timeouts the shared ones did not cover
fix(e2e): repair the import my own sweep broke in scroll-interaction
test(e2e): scale the per-call timeouts the shared ones did not cover
fix(e2e): repair the import my own sweep broke in scroll-interaction
test(e2e): scale the per-call timeouts the shared ones did not cover (#…
brentyates-swx commentedon Sep 22, 2026
Datapoint for the "varies by machine" half of this issue
While diagnosing #26 I ran this suite locally and got wildly inconsistent results — a 30-test run degrading to 7 tests in 17.4 minutes, exiting non-zero, with flake counts swinging between runs of identical code.
The cause was the machine, not the suite: load average was 90 (other sessions running on the same box). Playwright with 5–8 workers, each starting a Vite dev server and a Chromium, does not degrade gracefully at that load; it degrades into exactly the "environment-sensitive, varies by run" signature this issue describes.
Two practical consequences:
Local flake measurements on a shared box are not evidence. I quoted a "28 of 88 flaky" figure on e2e: session-todo spec fails on all three attempts within a run, then passes cleanly on later runs #26 from such a run and have withdrawn it. Anything measured here describes contention, not the code.
uptimebefore drawing conclusions is the cheap guard, and I did not do it.A before/after flake comparison needs a dedicated runner. CI is the only environment where the comparison means anything, which is why test(e2e): serve the v2 catalog routes from the mock server #37 deliberately makes no flake claim and leaves its own CI run as the first real datapoint.
This does not explain the CI-side failures — CI runners are not at load 90 — so it does not close anything here. But it does mean this issue's "and machine" clause is at least partly about where the suite is run rather than only about the tests, and that any future triage of it should record machine load alongside the failure.
Separately, one of the specs named in this issue now has a concrete non-timing root cause: see #26 and #37.
session-timeline-transport.spec.tswas not it — that one failed once and passed on retry with no app error, so it remains unexplained and should not be closed by #37.brentyates-swx commentedon Sep 23, 2026
Closing as not planned, with #26 and #39, for the same reason: this is upstream's test suite and this fork does not maintain it.
.github/workflows/test.ymlno longer gates merges on the app e2e suite, so its environment-sensitivity cannot block our feature work.One datapoint worth keeping if anyone revisits it: my local measurements here were void because this machine was at load average 90 from other sessions — a 30-test run degraded to 7 tests in 17.4 minutes and exited non-zero. I withdrew a "28 of 88 flaky" figure for that reason. Any future triage of this should record machine load alongside the failure, and prefer a dedicated runner.