Repository navigation
test(browser): survive the live-stream restart a page-clock jump causes - #3442
Conversation
…120 s jump The remote-failure history test pauses the page clock, freezes wall time, and runs 120 s of browser time. The live broker stream arms its 45 s heartbeat on the page clock while the broker's 15 s keepalive runs on real time, so the jump always expires the heartbeat and restarts the stream. Two flakes followed: the test published its signed replies before the restarted stream had re-issued a channel REQ, so the policy relay rejected the publish; and on a slow jump the still-frozen wall clock drifted more than 60 s from the broker, so read-state signing answered 400. Re-anchor the page wall clock to real time right after resuming, so signed read-state events follow the broker's clock again, and poll for the primary/alpha route before publishing, mirroring the readiness wait the test already does at startup. Verified with `bin/pnpm test:browser:ci --project webkit --no-deps --workers=1 --repeat-each=8 --reporter=list tests/browser/agent-outcomes.spec.mjs -g "a remote failure shares"`: 8/8 passed locally (17.8–20.2 s each). Co-Authored-By: Claude Fable 5.1 <[email protected]> Signed-off-by: mtoohey <[email protected]> Signed-off-by: Matt Toohey <[email protected]>
The relay-recovery journey fast-forwards the page clock 30 s twice. The live stream arms its 45 s heartbeat on the page clock while the broker's 15 s keepalive runs on real time, so the second jump expires the heartbeat and restarts the stream. In the same jump, session recovery's scheduled retry posts /stream-retry with the old stream id; the broker has already handled the SSE close and answers 404 "Live stream no longer available". The app drops the error for a superseded generation, but the browser still logs "Failed to load resource … 404", and the fixture only excused retired-stream 404s for /stream-observer and /stream-priority, so the unexplained-console-error check failed the test. Widen the fixture's match to the six broker control routes that answer 404 once a stream is retired — observer, priority, retry, interests, artifact-sync and artifact-sync-close — and keep the exact `retiredStreams.has(streamId)` guard, so a 404 for a live or unknown stream still fails. Presence controls answer 200 for a missing stream and stay out of the list. Verified with `bin/pnpm test:browser:ci --project webkit --no-deps --workers=1 --repeat-each=8 --reporter=list tests/browser/relay-recovery.spec.mjs tests/browser/relay-refusal.spec.mjs -g "relay reads recover automatically|an upload is the first request"`: 16/16 passed locally (relay-recovery 4.2–4.7 s, relay-refusal ~2.0 s). The late /stream-retry 404 did not reproduce locally — no retired controls were recorded in evidence.json — so the widened branch is only exercised under CI timing (Flake check lane). Co-Authored-By: Claude Fable 5.1 <[email protected]> Signed-off-by: mtoohey <[email protected]> Signed-off-by: Matt Toohey <[email protected]>
kazemek21
left a comment
There was a problem hiding this comment.
Kyle reviewed this PR and the Nexus review-tracker notes and approved with no comments.
Posted by Codex (gpt-6.1-sol) on behalf of kylez.
wesbillman
left a comment
There was a problem hiding this comment.
🤖 Star Lord’s automated source review via Wes’s account, with independent Rocket, Mantis and Groot coverage.
Reviewed head 3466600ae887595245e398ec26d27d0e6acadcbf against base/merge-base 7f03764beb8eb0ddcb4ff9ec0b8bda3e07699716. No actionable findings.
The wall-clock re-anchor preserves the elapsed-time assertions, and the route poll establishes a channel REQ before signed replies are published. The six-route 404 allowance retains the exact retired-stream-ID check and one-response/one-console-error matching; live or unknown stream IDs still fail the fixture. Existing keyboard, disclosure, persistence and geometry assertions remain intact.
Validation: source review only; we executed no tests or app code. Existing CI run 43354804087 passed, including both engines’ flake jobs. Its tested merge 1b209bfad093558dd79226f6181f72f25f03296d combines the exact head/base above. Inspected shard-1 artifacts contain two passing repetitions of each affected journey per engine, with no retired controls recorded in those cases. These samples do not exercise the widened 404 branch or reproduce the reported unusually slow clock jump; the route poll also does not establish a particular replacement generation. No claim that all timing races are eliminated.
…y-unreads * origin/main: (24 commits) [1/2] feat(agents): add a recommended agent setup (core) (#3344) test(browser): survive the live-stream restart a page-clock jump causes (#3442) chore(renovate): hold rusqlite below 0.41 for mesh-llm (#3438) test(browser): serve the diffs and CLI reload fixtures outside the navigation budget (#3445) test(browser): play the draft clip at rate 0 instead of looping it before the Play click (#3446) fix(status): reopen a status whose preset deadlines coincide as Today (#3449) fix(messages): tolerate a cancelled reaction-name chunk load (#3450) fix(pairing): tolerate /proc/net/tcp rows printed twice or missed (#3451) test(browser): wait on the signal each test asserts in local-agent-activity and message-navigation (#3444) Expose native channel headings to Solo sources (#3523) test(browser-host): run the 100,000-channel broker scale check alone in the measurements lane (#3447) fix(solo): keep every contributed destination action (#3516) fix(identity): report an issuer that keeps rate-limiting an agent as a limit (#2365) fix(identity): let a stuck at-launch sign-in copy its link (#2453) fix(profiles): open avatar previews on hover in compact density (#3510) feat(solo): smooth chat recency reordering (#3502) Reject self-mentions that cannot wake another session (#3513) feat(status): offer don't clear for every status (#3515) Let plugin panels name their tab with panels.useTabTitle (#3251) fix(agents): report a conversation wake that fails before its turn starts (#2192) ... # Conflicts: # tests/browser/message-management.spec.mjs
* origin/main: (27 commits) [2/2] feat(agents): offer Automatic (Recommended) in Create agent and Identity (#3345) fix(solo): preserve and resume unsent chat drafts (#3525) fix(builderlab): give refused app sign-ins and Identity links a route to Organization sign-in (#3078) fix(agents): say why Import an agent can't import here, and link Manage plugins (#3073) [1/2] feat(agents): add a recommended agent setup (core) (#3344) test(browser): survive the live-stream restart a page-clock jump causes (#3442) chore(renovate): hold rusqlite below 0.41 for mesh-llm (#3438) test(browser): serve the diffs and CLI reload fixtures outside the navigation budget (#3445) test(browser): play the draft clip at rate 0 instead of looping it before the Play click (#3446) fix(status): reopen a status whose preset deadlines coincide as Today (#3449) fix(messages): tolerate a cancelled reaction-name chunk load (#3450) fix(pairing): tolerate /proc/net/tcp rows printed twice or missed (#3451) test(browser): wait on the signal each test asserts in local-agent-activity and message-navigation (#3444) Expose native channel headings to Solo sources (#3523) test(browser-host): run the 100,000-channel broker scale check alone in the measurements lane (#3447) fix(solo): keep every contributed destination action (#3516) fix(identity): report an issuer that keeps rate-limiting an agent as a limit (#2365) fix(identity): let a stuck at-launch sign-in copy its link (#2453) fix(profiles): open avatar previews on hover in compact density (#3510) feat(solo): smooth chat recency reordering (#3502) ... # Conflicts: # src-tauri/src/federated_identity/agent/tests.rs # src-tauri/src/federated_identity/mod.rs # src-tauri/src/federated_identity/protocol.rs
* origin/main: (133 commits) Expose ordinary chats without workspace headings (block#3530) [2/2] feat(agents): offer Automatic (Recommended) in Create agent and Identity (block#3345) fix(solo): preserve and resume unsent chat drafts (block#3525) fix(builderlab): give refused app sign-ins and Identity links a route to Organization sign-in (block#3078) fix(agents): say why Import an agent can't import here, and link Manage plugins (block#3073) [1/2] feat(agents): add a recommended agent setup (core) (block#3344) test(browser): survive the live-stream restart a page-clock jump causes (block#3442) chore(renovate): hold rusqlite below 0.41 for mesh-llm (block#3438) test(browser): serve the diffs and CLI reload fixtures outside the navigation budget (block#3445) test(browser): play the draft clip at rate 0 instead of looping it before the Play click (block#3446) fix(status): reopen a status whose preset deadlines coincide as Today (block#3449) fix(messages): tolerate a cancelled reaction-name chunk load (block#3450) fix(pairing): tolerate /proc/net/tcp rows printed twice or missed (block#3451) test(browser): wait on the signal each test asserts in local-agent-activity and message-navigation (block#3444) Expose native channel headings to Solo sources (block#3523) test(browser-host): run the 100,000-channel broker scale check alone in the measurements lane (block#3447) fix(solo): keep every contributed destination action (block#3516) fix(identity): report an issuer that keeps rate-limiting an agent as a limit (block#2365) fix(identity): let a stuck at-launch sign-in copy its link (block#2453) fix(profiles): open avatar previews on hover in compact density (block#3510) ... Signed-off-by: Joah Gerstenberg <[email protected]> # Conflicts: # src/bundled/codex/prompt.test.ts # src/bundled/goose/runtime.ts
Why
Two browser journeys fail intermittently in CI for the same underlying reason. Both jump the page clock far enough to expire the live stream's heartbeat, and the stream restart that follows was not accounted for.
agent-outcomes.spec.mjs, "a remote failure shares its expandable history and preserves its message anchor"relay-recovery.spec.mjs, "relay reads recover automatically after the network returns"Root cause
The live broker stream arms a 45 s heartbeat on the page clock. The broker's 15 s keepalive runs on real time. A page-clock jump longer than 45 s therefore always expires the heartbeat, and the page aborts and restarts its stream.
presence.spec.mjsalready records this hazard.agent-outcomespauses the page clock, freezes wall time, and runs 120 s of browser time. Two failures follow from the restart:relay-recoveryfast-forwards the page clock 30 s twice. The second jump expires the heartbeat. In the same jump, session recovery's scheduled retry posts/stream-retrywith the old stream id. The broker has already handled the stream close and answers 404 "Live stream no longer available". The app drops the error for a superseded generation, but the browser still logs "Failed to load resource … 404". The fixture only excused retired-stream 404s for/stream-observerand/stream-priority, so the unexplained-console-error check failed the test. The trace shows the fast-forward, the 404 ten milliseconds later, and the new stream 0.7 s after that.Neither is a product defect. The restart only happens under a fake-clock jump, and the app already ignores the late control response.
Fix
agent-outcomes.spec.mjs: re-anchor the page wall clock to real time right after resuming, so signed read-state events follow the broker's clock again. Then poll for the primary/alpha route before publishing, mirroring the readiness wait the test already does at startup.fixture.mjs: widen the retired-stream 404 match to the six broker control routes that answer 404 once a stream is retired: observer, priority, retry, interests, artifact-sync and artifact-sync-close. The exactretiredStreams.has(streamId)guard stays, so a 404 for a live or unknown stream still fails. Presence controls answer 200 for a missing stream and stay out of the list.No timeouts, retries or assertions were relaxed. No browser cases were added or removed.
Evidence
Local runs on Linux, WebKit,
bin/pnpm test:browser:ci --project webkit --no-deps --workers=1 --repeat-each=8 --reporter=list.agent-outcomes.spec.mjs -g "a remote failure shares"relay-recovery.spec.mjs relay-refusal.spec.mjs -g "relay reads recover automatically|an upload is the first request"Not reproduced, not run
/stream-retry404 did not reproduce locally. No retired controls were recorded in the local evidence, so the widened fixture branch is only exercised under CI timing. The Flake check lane on this PR is the real test of it.agent-outcomestest also passed 8/8 locally, with two or three stream restarts during each jump. The local run guards against regression rather than reproducing the CI ordering.buzz-review-completedis not set.Overlap with open PRs
Several open PRs also edit
tests/browser/fixture.mjs(for example #3332, #3239, #3186 and #3093). None touches the retired-stream match, and a trial merge of each against this change is clean.🤖 Generated with Claude Code