Skip to content

test(browser): survive the live-stream restart a page-clock jump causes - #3442

Merged
matt2e merged 2 commits into
mainfrom
mtoohey/flake-clock-jump-heartbeat
Nov 14, 2026
Merged

matt2e merged 2 commits into
mainfrom
mtoohey/flake-clock-jump-heartbeat

Conversation

@matt2e

@matt2e matt2e commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Why

Two browser journeys fail intermittently in CI for the same underlying reason. Both jump the page clock far enough to expire the live stream's heartbeat, and the stream restart that follows was not accounted for.

Test Sightings
agent-outcomes.spec.mjs, "a remote failure shares its expandable history and preserves its message anchor" WebKit 1/18, Browser journeys: 10 Nov, main, 10 Nov, PR, 11 Nov, PR, 11 Nov, main
relay-recovery.spec.mjs, "relay reads recover automatically after the network returns" WebKit 5/18, Browser journeys: 10 Nov, PR, passed on re-run

Root cause

The live broker stream arms a 45 s heartbeat on the page clock. The broker's 15 s keepalive runs on real time. A page-clock jump longer than 45 s therefore always expires the heartbeat, and the page aborts and restarts its stream. presence.spec.mjs already records this hazard.

agent-outcomes pauses the page clock, freezes wall time, and runs 120 s of browser time. Two failures follow from the restart:

  • In three of the four jobs the test published its signed replies before the restarted stream had re-issued a channel REQ. The policy relay rejected the publish with "event must traverse an explicit production channel REQ". In each trace the publish came about 0.9 s after the clock resumed, and the new stream opened 0.2 to 0.4 s later.
  • In the fourth job (11 Nov, main) the 120 s jump took 85.6 s of real time instead of the usual 4 to 12 s. The still-frozen wall clock drifted more than 60 s from the broker, so read-state signing answered 400.

relay-recovery fast-forwards the page clock 30 s twice. The second jump expires the heartbeat. In the same jump, session recovery's scheduled retry posts /stream-retry with the old stream id. The broker has already handled the stream close and answers 404 "Live stream no longer available". The app drops the error for a superseded generation, but the browser still logs "Failed to load resource … 404". The fixture only excused retired-stream 404s for /stream-observer and /stream-priority, so the unexplained-console-error check failed the test. The trace shows the fast-forward, the 404 ten milliseconds later, and the new stream 0.7 s after that.

Neither is a product defect. The restart only happens under a fake-clock jump, and the app already ignores the late control response.

Fix

  • agent-outcomes.spec.mjs: re-anchor the page wall clock to real time right after resuming, so signed read-state events follow the broker's clock again. Then poll for the primary/alpha route before publishing, mirroring the readiness wait the test already does at startup.
  • fixture.mjs: widen the retired-stream 404 match to the six broker control routes that answer 404 once a stream is retired: observer, priority, retry, interests, artifact-sync and artifact-sync-close. The exact retiredStreams.has(streamId) guard stays, so a 404 for a live or unknown stream still fails. Presence controls answer 200 for a missing stream and stay out of the list.

No timeouts, retries or assertions were relaxed. No browser cases were added or removed.

Evidence

Local runs on Linux, WebKit, bin/pnpm test:browser:ci --project webkit --no-deps --workers=1 --repeat-each=8 --reporter=list.

Scope Result
agent-outcomes.spec.mjs -g "a remote failure shares" 8/8 passed, 17.8 to 20.2 s each
relay-recovery.spec.mjs relay-refusal.spec.mjs -g "relay reads recover automatically|an upload is the first request" 16/16 passed

Not reproduced, not run

  • The late /stream-retry 404 did not reproduce locally. No retired controls were recorded in the local evidence, so the widened fixture branch is only exercised under CI timing. The Flake check lane on this PR is the real test of it.
  • The unchanged agent-outcomes test also passed 8/8 locally, with two or three stream restarts during each jump. The local run guards against regression rather than reproducing the CI ordering.
  • The read-state re-anchor addresses a failure seen once, when the jump ran unusually slowly. That slowness was not reproduced.
  • Chromium was not run for these two changes.
  • No human has tested this yet, so buzz-review-completed is not set.

Overlap with open PRs

Several open PRs also edit tests/browser/fixture.mjs (for example #3332, #3239, #3186 and #3093). None touches the retired-stream match, and a trial merge of each against this change is clean.

🤖 Generated with Claude Code

matt2e and others added 2 commits November 13, 2026 13:39
…120 s jump

The remote-failure history test pauses the page clock, freezes wall time,
and runs 120 s of browser time. The live broker stream arms its 45 s
heartbeat on the page clock while the broker's 15 s keepalive runs on real
time, so the jump always expires the heartbeat and restarts the stream.
Two flakes followed: the test published its signed replies before the
restarted stream had re-issued a channel REQ, so the policy relay rejected
the publish; and on a slow jump the still-frozen wall clock drifted more
than 60 s from the broker, so read-state signing answered 400.

Re-anchor the page wall clock to real time right after resuming, so signed
read-state events follow the broker's clock again, and poll for the
primary/alpha route before publishing, mirroring the readiness wait the
test already does at startup.

Verified with `bin/pnpm test:browser:ci --project webkit --no-deps
--workers=1 --repeat-each=8 --reporter=list
tests/browser/agent-outcomes.spec.mjs -g "a remote failure shares"`:
8/8 passed locally (17.8–20.2 s each).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: mtoohey <[email protected]>
Signed-off-by: Matt Toohey <[email protected]>
The relay-recovery journey fast-forwards the page clock 30 s twice. The
live stream arms its 45 s heartbeat on the page clock while the broker's
15 s keepalive runs on real time, so the second jump expires the heartbeat
and restarts the stream. In the same jump, session recovery's scheduled
retry posts /stream-retry with the old stream id; the broker has already
handled the SSE close and answers 404 "Live stream no longer available".
The app drops the error for a superseded generation, but the browser still
logs "Failed to load resource … 404", and the fixture only excused
retired-stream 404s for /stream-observer and /stream-priority, so the
unexplained-console-error check failed the test.

Widen the fixture's match to the six broker control routes that answer 404
once a stream is retired — observer, priority, retry, interests,
artifact-sync and artifact-sync-close — and keep the exact
`retiredStreams.has(streamId)` guard, so a 404 for a live or unknown stream
still fails. Presence controls answer 200 for a missing stream and stay
out of the list.

Verified with `bin/pnpm test:browser:ci --project webkit --no-deps
--workers=1 --repeat-each=8 --reporter=list
tests/browser/relay-recovery.spec.mjs tests/browser/relay-refusal.spec.mjs
-g "relay reads recover automatically|an upload is the first request"`:
16/16 passed locally (relay-recovery 4.2–4.7 s, relay-refusal ~2.0 s). The
late /stream-retry 404 did not reproduce locally — no retired controls were
recorded in evidence.json — so the widened branch is only exercised under
CI timing (Flake check lane).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: mtoohey <[email protected]>
Signed-off-by: Matt Toohey <[email protected]>
@matt2e
matt2e requested a review from a team as a code owner November 13, 2026 02:49

@kazemek21 kazemek21 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kyle reviewed this PR and the Nexus review-tracker notes and approved with no comments.

Posted by Codex (gpt-6.1-sol) on behalf of kylez.

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Star Lord’s automated source review via Wes’s account, with independent Rocket, Mantis and Groot coverage.

Reviewed head 3466600ae887595245e398ec26d27d0e6acadcbf against base/merge-base 7f03764beb8eb0ddcb4ff9ec0b8bda3e07699716. No actionable findings.

The wall-clock re-anchor preserves the elapsed-time assertions, and the route poll establishes a channel REQ before signed replies are published. The six-route 404 allowance retains the exact retired-stream-ID check and one-response/one-console-error matching; live or unknown stream IDs still fail the fixture. Existing keyboard, disclosure, persistence and geometry assertions remain intact.

Validation: source review only; we executed no tests or app code. Existing CI run 43354804087 passed, including both engines’ flake jobs. Its tested merge 1b209bfad093558dd79226f6181f72f25f03296d combines the exact head/base above. Inspected shard-1 artifacts contain two passing repetitions of each affected journey per engine, with no retired controls recorded in those cases. These samples do not exercise the widened 404 branch or reproduce the reported unusually slow clock jump; the route poll also does not establish a particular replacement generation. No claim that all timing races are eliminated.

@matt2e
matt2e merged commit d3a67a7 into main Nov 14, 2026
63 checks passed
@matt2e
matt2e deleted the mtoohey/flake-clock-jump-heartbeat branch November 14, 2026 00:57
wpfleger96 added a commit that referenced this pull request Nov 14, 2026
…y-unreads

* origin/main: (24 commits)
  [1/2] feat(agents): add a recommended agent setup (core) (#3344)
  test(browser): survive the live-stream restart a page-clock jump causes (#3442)
  chore(renovate): hold rusqlite below 0.41 for mesh-llm (#3438)
  test(browser): serve the diffs and CLI reload fixtures outside the navigation budget (#3445)
  test(browser): play the draft clip at rate 0 instead of looping it before the Play click (#3446)
  fix(status): reopen a status whose preset deadlines coincide as Today (#3449)
  fix(messages): tolerate a cancelled reaction-name chunk load (#3450)
  fix(pairing): tolerate /proc/net/tcp rows printed twice or missed (#3451)
  test(browser): wait on the signal each test asserts in local-agent-activity and message-navigation (#3444)
  Expose native channel headings to Solo sources (#3523)
  test(browser-host): run the 100,000-channel broker scale check alone in the measurements lane (#3447)
  fix(solo): keep every contributed destination action (#3516)
  fix(identity): report an issuer that keeps rate-limiting an agent as a limit (#2365)
  fix(identity): let a stuck at-launch sign-in copy its link (#2453)
  fix(profiles): open avatar previews on hover in compact density (#3510)
  feat(solo): smooth chat recency reordering (#3502)
  Reject self-mentions that cannot wake another session (#3513)
  feat(status): offer don't clear for every status (#3515)
  Let plugin panels name their tab with panels.useTabTitle (#3251)
  fix(agents): report a conversation wake that fails before its turn starts (#2192)
  ...

# Conflicts:
#	tests/browser/message-management.spec.mjs
wpfleger96 added a commit that referenced this pull request Nov 14, 2026
* origin/main: (27 commits)
  [2/2] feat(agents): offer Automatic (Recommended) in Create agent and Identity (#3345)
  fix(solo): preserve and resume unsent chat drafts (#3525)
  fix(builderlab): give refused app sign-ins and Identity links a route to Organization sign-in (#3078)
  fix(agents): say why Import an agent can't import here, and link Manage plugins (#3073)
  [1/2] feat(agents): add a recommended agent setup (core) (#3344)
  test(browser): survive the live-stream restart a page-clock jump causes (#3442)
  chore(renovate): hold rusqlite below 0.41 for mesh-llm (#3438)
  test(browser): serve the diffs and CLI reload fixtures outside the navigation budget (#3445)
  test(browser): play the draft clip at rate 0 instead of looping it before the Play click (#3446)
  fix(status): reopen a status whose preset deadlines coincide as Today (#3449)
  fix(messages): tolerate a cancelled reaction-name chunk load (#3450)
  fix(pairing): tolerate /proc/net/tcp rows printed twice or missed (#3451)
  test(browser): wait on the signal each test asserts in local-agent-activity and message-navigation (#3444)
  Expose native channel headings to Solo sources (#3523)
  test(browser-host): run the 100,000-channel broker scale check alone in the measurements lane (#3447)
  fix(solo): keep every contributed destination action (#3516)
  fix(identity): report an issuer that keeps rate-limiting an agent as a limit (#2365)
  fix(identity): let a stuck at-launch sign-in copy its link (#2453)
  fix(profiles): open avatar previews on hover in compact density (#3510)
  feat(solo): smooth chat recency reordering (#3502)
  ...

# Conflicts:
#	src-tauri/src/federated_identity/agent/tests.rs
#	src-tauri/src/federated_identity/mod.rs
#	src-tauri/src/federated_identity/protocol.rs
joahg added a commit to joahg/buzz-app that referenced this pull request Nov 14, 2026
* origin/main: (133 commits)
  Expose ordinary chats without workspace headings (block#3530)
  [2/2] feat(agents): offer Automatic (Recommended) in Create agent and Identity (block#3345)
  fix(solo): preserve and resume unsent chat drafts (block#3525)
  fix(builderlab): give refused app sign-ins and Identity links a route to Organization sign-in (block#3078)
  fix(agents): say why Import an agent can't import here, and link Manage plugins (block#3073)
  [1/2] feat(agents): add a recommended agent setup (core) (block#3344)
  test(browser): survive the live-stream restart a page-clock jump causes (block#3442)
  chore(renovate): hold rusqlite below 0.41 for mesh-llm (block#3438)
  test(browser): serve the diffs and CLI reload fixtures outside the navigation budget (block#3445)
  test(browser): play the draft clip at rate 0 instead of looping it before the Play click (block#3446)
  fix(status): reopen a status whose preset deadlines coincide as Today (block#3449)
  fix(messages): tolerate a cancelled reaction-name chunk load (block#3450)
  fix(pairing): tolerate /proc/net/tcp rows printed twice or missed (block#3451)
  test(browser): wait on the signal each test asserts in local-agent-activity and message-navigation (block#3444)
  Expose native channel headings to Solo sources (block#3523)
  test(browser-host): run the 100,000-channel broker scale check alone in the measurements lane (block#3447)
  fix(solo): keep every contributed destination action (block#3516)
  fix(identity): report an issuer that keeps rate-limiting an agent as a limit (block#2365)
  fix(identity): let a stuck at-launch sign-in copy its link (block#2453)
  fix(profiles): open avatar previews on hover in compact density (block#3510)
  ...

Signed-off-by: Joah Gerstenberg <[email protected]>

# Conflicts:
#	src/bundled/codex/prompt.test.ts
#	src/bundled/goose/runtime.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants