You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e6bcd02
Browse filesBrowse the repository at this point in the historyBrowse files
test(plugin): skip openai-ws idle-pool prune test, known-flaky under normal load (TKT-349)
Failed 3x across this PR's own CI reruns, unrelated to the diff each time --
real setTimeout-based idle-connection-pruning race, timing-sensitive under
this host's normal ~9 load average. Retracts the earlier "wait for a quiet
window" policy (that threshold was set from one settling sample, not this
host's actual normal operating range) in favor of a bounded one: rerun once
regardless of load, skip-with-comment a spec that repeats across a PR's own
runs. Filed as feedback anomalyco#180.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JJQrqYMv43CrdbjpTT7884
pruning timer, `ModelsDev Service`'s cache-fetch hitting its 30000ms timeout at 30000.20ms),
490
-
none touching the PR's own diff, none repeating across attempts. The discriminator: host load
491
-
average was ~9.4 across the failing window and had settled to ~4 by the time a clean rerun
492
-
passed. **N reruns inside one loaded window are ONE sample of the window, not N independent
493
-
samples — back-to-back reruns cannot distinguish flake from regression while the load stays
494
-
high.** Check `uptime` before burning a rerun, same rule as above but gated on load instead of
495
-
spec identity: one rerun in a quiet window (1-minute load average below ~6) is a real second
496
-
sample; three reruns inside a loaded window are not three chances to see through a regression,
497
-
they are three chances to see three different timers miss under contention. Same-spec-3-
498
-
consecutive-in-a-quiet-window still means real regression, unchanged from the e2e rule above.
499
-
Load-sensitive deflake candidates (the 3 specs above) filed as feedback for whoever picks up
500
-
hardening them — fake-clock (`bun:test`'s `setSystemTime`/a controlled timer, rather than a
501
-
real `setTimeout`/timeout race) is the fix shape that removes the host-load dependency entirely.
490
+
none touching the PR's own diff. An earlier version of this entry proposed a "wait for host load
491
+
under ~6" precondition before rerunning — **retracted**: checked directly, `load average ~9` is
492
+
this host's *normal* operating state while two fleets work (vLLM inference serving both fleets,
493
+
concurrent test/worker sessions), not a spike to wait out. A threshold set from one settling
494
+
sample is the same unmeasured-threshold mistake as elsewhere in this ledger — don't repeat the
495
+
shape even when the specific number looks plausible. **Corrected, bounded policy:** unit reruns
496
+
take the host as it is, no load precondition — fire one rerun of the failed job whenever the
497
+
runner is free. If a spec fails that has already failed in a **prior run of the same PR**
498
+
(`plugin.openai.ws-pool` reached 3 failures across TKT-349's own reruns), treat it as
499
+
known-flaky-under-normal-load: skip it with `test.skip` and a comment linking the deflake
500
+
feedback item, and add a row to the "Resolved CI reds" table below (Sean's standing directive:
501
+
cheaper to skip a particular known unit test with a comment than to keep re-running around it).
502
+
A spec with only one failure across a PR's runs stays live — no skip on a single occurrence.
503
+
Fix shape for whoever picks up hardening these: fake-clock (`bun:test`'s `setSystemTime`/a
504
+
controlled timer, rather than a real `setTimeout`/timeout race), which removes the host-load
505
+
dependency entirely instead of tolerating
506
+
it.
502
507
-**`nix-eval`:** disabled (auto-trigger removed, see "Blacksmith runner sweep" below) rather
503
508
than gated or merely observational — Nix packaging validity is not this fork's concern at all,
504
509
so there is nothing to read a rule against.
@@ -522,6 +527,7 @@ like; nothing here is a currently-live gate exception.
522
527
| `unit (linux)` — 7 additional `run-process.test.ts` tests, briefly, only on TKT-315's own PR: clustered 700-800ms over the harness's 30\_000ms default `timeoutMs` | TKT-315 added `ProjectV2.node`/`SessionGoal.node`/`SessionLedger.node` to `app-runtime.ts`'s CLI/TUI `AppLayer` (see "Adding a new global `.node`?" above — all three traced reachable, not removable). None do eager I/O, but building 3 more global nodes has nonzero per-process construction cost; invisible in isolated timing (8 runs each, branch vs. dev, fully overlapping) but real under `run-process.test.ts`'s 13 concurrent `cliIt.concurrent()` subprocess spawns (6x full-file runs, non-overlapping ranges). | **Interim `timeoutMs` widen shipped in the same PR** (30\_000 → 45\_000 on the 7 affected tests only, each commented, feedback #150) — the real fix (lazy node construction so a CLI invocation that never touches project/goal/ledger data pays nothing for them) is still open, tracked in #150. This is the one row here that is not fully closed — it is an accepted, narrow, linked interim, not a currently-observed red. |
523
528
|`packages/opencode test:httpapi`, `mode=effect` only — `v2.session.goal.get`, "a session with no goal set should report no data" | Was masked by a crash (see the `SessionGoal.node`/`SessionLedger.node` wiring gap above) until TKT-315 fixed the wiring and let the route's assertion actually run; the assertion itself then failed, a real bug in TKT-317's goal-get response path (filed as feedback #145). |**Fixed** ([#10](https://github.com/Draugur-AI/opencode/pull/10), Gemma) — confirmed dead on `dev@441ad185`: `test:httpapi --mode effect` went from 227/0 (crash-masked) to 226/1 (this assertion, post-TKT-315) to 229/0 (post-#10, three new scenarios added along the way, zero fail). No exception needed in the simplified gate above. |
524
529
| `e2e (windows)`, first run only — apparent ~40 `e2e/regression/*.spec.ts` failures | **Misread, not a real symptom** — the ~40 count was a grep artifact (spec-file-path occurrences anywhere in the log, which also matches Playwright's test-discovery listing) mistaken for a failure list. The real first-run result was 2 failed / 87 passed. A subsequent run showed `e2e (linux)` — clean on its first two runs — fail 1 spec too, a *different* spec. Both platforms show the same class of ordinary, low-rate E2E flakiness, pre-existing and simply never observed before (e2e never ran on either platform pre-TKT-336, both queued forever on blacksmith). | **Not a red to resolve — moved out of the gate entirely, symmetrically on both platforms** (Ethan's ruling: stabilizing genuine Playwright flakiness is a different body of work than CI hygiene, belongs to the milestone-3 real-browser-matrix slice). See "Observational checks" above for the deterministic reading rule that replaces the skip. Feedback #156 corrected in place rather than superseded — the correction is part of its own record. |
530
+
|`unit (linux)` — `plugin.openai.ws-pool`, "prunes idle websocket connections after completed responses" | Real `setTimeout`-based idle-connection-pruning race, timing-sensitive under normal host load (this host runs ~9 load average continuously serving two fleets' inference/tests — not a spike). Failed 3 times across TKT-349's own CI reruns of one unchanged commit, unrelated to that PR's diff. |**Skipped** (TKT-349, feedback #180): `test.skip` with a comment linking the feedback item. Un-skip condition: feedback #180 is triaged — fake-clock (`setSystemTime`/a controlled timer) replacing the real timer removes the flake at its root rather than just tolerating it. |
0 commit comments