Skip to content

Commit c8335eb

Browse files
committed
fix(webview): log terminal recovery stops after retry warnings
1 parent 82c2a47 commit c8335eb

6 files changed

Lines changed: 148 additions & 30 deletions

File tree

‎apps/vscode-e2e/src/visual/heartbeat.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ The message-port interceptor follows the pinned VS Code implementation. Check it
1717

1818
A gray sidebar can persist after renderer termination even when the watchdog assigns fresh HTML. HTML delivery alone cannot restart a dead renderer. Recovery now waits ten seconds for a heartbeat after assigning HTML, then recreates the affected host if the renderer has not responded. Sidebar recovery uses the generated view-removal and focus commands after checking their availability; tab recovery replaces its panel while retaining its provider. The crash test covers the real sidebar path. Unit tests cover tab replacement and task ownership.
1919

20-
A recovery attempt includes HTML-generation errors. Three attempts exhaust the budget; a startup heartbeat alone does not reset it. Periodic heartbeats must span a 60-second healthy window to replenish the budget. Exhaustion or unavailable/failed scoped host recovery logs once and offers a non-modal Reload Window or Show Logs action. A sidebar replacement that arrives after the ten-second wait retains the incident budget and displayed task. A failed sidebar focus command clears the pending marker so a later manual open starts normally. A sidebar or tab that fails initialization after publication shows the recovery warning and keeps its task and a watchdog for bounded retries. Failure handling lives in the shared view resolver, including sidebar callbacks invoked independently by VS Code. A published sidebar that is still initializing after the ten-second wait keeps its remaining attempts. The warning does not consume the remaining attempts. Duplicate warnings are suppressed until heartbeats resume or a user-opened view resets the notification state. Only sustained healthy heartbeats replenish the incident budget. Window reload is never automatic. A late Reload Window action from an old notification is ignored after liveness returns. Show Logs still opens the output channel.
20+
A recovery attempt includes HTML-generation errors. Three attempts exhaust the budget; a startup heartbeat alone does not reset it. Periodic heartbeats must span a 60-second healthy window to replenish the budget. Exhaustion or unavailable/failed scoped host recovery logs once and offers a non-modal Reload Window or Show Logs action. A sidebar replacement that arrives after the ten-second wait retains the incident budget and displayed task. A failed sidebar focus command clears the pending marker so a later manual open starts normally. A sidebar or tab that fails initialization after publication shows the recovery warning and keeps its task and a watchdog for bounded retries. Failure handling lives in the shared view resolver, including sidebar callbacks invoked independently by VS Code. A published sidebar that is still initializing after the ten-second wait keeps its remaining attempts. The warning does not consume the remaining attempts. Duplicate warnings are suppressed until heartbeats resume or a user-opened view resets the notification state. A later terminal failure still logs its final stop reason once, independently of warning deduplication. Subsequent watchdog ticks do not repeat the stop log. Only sustained healthy heartbeats replenish the incident budget. Window reload is never automatic. A late Reload Window action from an old notification is ignored after liveness returns. Show Logs still opens the output channel.
2121

2222
Keep exact timing boundaries, individual post-await guards, and reload-budget assertions in provider unit tests. Exercise OS minimize and sleep/resume behavior manually; the Electron scenarios do not reproduce OS suspension or its scheduling order.
2323

‎docs/architecture/webview-recovery-model.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Code QA runs both commands in its compile job. This is an independent webview pr
1111

1212
## Production connection
1313

14-
`src/core/webview/webviewRecoveryPolicy.ts` owns pure decisions used by both `ClineProvider` and the checker. These include attempt admission, recovery-incident preservation, resolve-failure classification, HTML commit eligibility, release of an async recovery owner, command-failure marker cleanup, failure accounting and notification deduplication, heartbeat health, and warning actions.
14+
`src/core/webview/webviewRecoveryPolicy.ts` owns pure decisions used by both `ClineProvider` and the checker. These include attempt admission, recovery-incident preservation, resolve-failure classification, HTML commit eligibility, release of an async recovery owner, command-failure marker cleanup, failure accounting, notification deduplication and terminal-stop logging, heartbeat health, and warning actions.
1515

1616
The provider still performs VS Code effects. It installs subscriptions, starts timers, assigns HTML, resolves views, and retains its task stack. The policy extraction does not move those effects into the model.
1717

@@ -41,15 +41,15 @@ The checker rejects:
4141
3. duplicate in-flight recovery or attempts exceeding the cap;
4242
4. a recovery replacement resetting its incident budget or being classified as normal startup;
4343
5. a current recovery resolve failure losing the retry disposition;
44-
6. retryable warnings consuming the remaining attempts, or duplicate notifications without a notification-state reset;
44+
6. retryable warnings consuming the remaining attempts, duplicate notifications, missing terminal-stop reasons after retry warnings, or repeated terminal-stop logs without an incident reset;
4545
7. a failed sidebar focus leaving a marker that contaminates a later manual reopen;
4646
8. a startup heartbeat replenishing attempts before sustained health;
4747
9. a stale Reload Window action reloading after a heartbeat or disposal; and
4848
10. a late Show Logs action being incorrectly suppressed.
4949

50-
Required actions and semantic witnesses prevent a vacuous pass. They include a valid HTML commit, each invalidated completion class, a replacement owner surviving old completion, a retryable initialization failure, repeated initialization failures reaching the cap, healthy budget refill, a published sidebar timing out with attempts remaining, late publication retaining an exhausted budget, and a fresh manual open after focus failure.
50+
Required actions and semantic witnesses prevent a vacuous pass. They include a valid HTML commit, each invalidated completion class, a replacement owner surviving old completion, a retryable initialization failure, repeated initialization failures reaching the cap, healthy budget refill, a published sidebar timing out with attempts remaining, late publication retaining an exhausted budget, and a fresh manual open after focus failure, and one terminal-stop log after an earlier retry warning.
5151

52-
Ten representative faulty policies are injected without editing source. Each must fail with its expected contract violation. Exploration truncation or an unrelated exception does not count as rejecting a regression.
52+
Twelve representative faulty policies are injected without editing source. Each must fail with its expected contract violation. Exploration truncation or an unrelated exception does not count as rejecting a regression.
5353

5454
## Effect and integration evidence
5555

‎scripts/check-webview-recovery.ts‎

Lines changed: 68 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,9 @@ interface Host {
148148
phase: Phase
149149
attempts: number
150150
reported: boolean
151+
stopLogged: boolean
152+
terminalLogs: number
153+
retryWarned: boolean
151154
busy: boolean
152155
pending: boolean
153156
view: "old" | "none" | "replacement"
@@ -167,6 +170,9 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
167170
phase: "idle",
168171
attempts: 0,
169172
reported: false,
173+
stopLogged: false,
174+
terminalLogs: 0,
175+
retryWarned: false,
170176
busy: false,
171177
pending: false,
172178
view: "old",
@@ -186,10 +192,16 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
186192
const steps: Step<Host>[] = []
187193
const add = (action: string, update: Partial<Host>) => steps.push({ action, state: { ...s, ...update } })
188194
const failure = (stop: boolean): Partial<Host> => {
189-
const next = p.recordRecoveryFailure({ attempts: s.attempts, reported: s.reported }, stop)
195+
const next = p.recordRecoveryFailure(
196+
{ attempts: s.attempts, reported: s.reported, stopLogged: s.stopLogged },
197+
stop,
198+
)
190199
return {
191200
attempts: next.attempts,
192201
reported: next.reported,
202+
stopLogged: next.stopLogged,
203+
terminalLogs: s.terminalLogs + (stop && next.log ? 1 : 0),
204+
retryWarned: s.retryWarned || (!stop && next.notify),
193205
fault:
194206
next.attempts !== (stop ? 3 : s.attempts) || next.notify === s.reported
195207
? "failure consumed retries or repeated a warning"
@@ -292,6 +304,9 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
292304
watchdog: true,
293305
initialized: true,
294306
reported: false,
307+
stopLogged: false,
308+
terminalLogs: 0,
309+
retryWarned: false,
295310
manualOpened: true,
296311
fault: recovering ? "manual open inherited a failed incident" : undefined,
297312
})
@@ -305,6 +320,9 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
305320
add("startup-heartbeat", {
306321
healthStarted: true,
307322
reported: false,
323+
stopLogged: false,
324+
terminalLogs: 0,
325+
retryWarned: false,
308326
attempts: health.attempts,
309327
fault: health.attempts !== s.attempts ? "one heartbeat refilled the budget" : undefined,
310328
})
@@ -318,6 +336,9 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
318336
healthReset: true,
319337
attempts: health.attempts,
320338
reported: false,
339+
stopLogged: false,
340+
terminalLogs: 0,
341+
retryWarned: false,
321342
fault: health.attempts !== 0 ? "healthy window failed to replenish attempts" : undefined,
322343
})
323344
}
@@ -326,10 +347,13 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
326347
},
327348
(s) => {
328349
assert(!s.fault, s.fault)
350+
assert(s.terminalLogs <= 1, "duplicate terminal stop log in one incident")
351+
assert.equal(s.stopLogged, s.terminalLogs === 1, "terminal stop log was suppressed")
329352
assert(Number.isInteger(s.attempts) && s.attempts >= 0 && s.attempts <= 3, "attempt budget")
330353
if (s.phase === "idle")
331354
assert(s.watchdog && !s.busy, "settled recovery lost watchdog or retained ownership")
332355
const marks: string[] = []
356+
if (s.retryWarned && s.stopLogged && s.terminalLogs === 1) marks.push("terminal-stop-after-retry-warning")
333357
if (s.phase === "idle" && !s.initialized && s.attempts === 1) marks.push("initialization-failure-retryable")
334358
if (s.phase === "stopped" && s.attempts === 3 && !s.initialized && s.view === "replacement")
335359
marks.push("failed-resolutions-bounded")
@@ -342,7 +366,12 @@ function hostCheck(kind: "sidebar" | "tab", p: Policy): void {
342366
return marks
343367
},
344368
)
345-
const required = ["initialization-failure-retryable", "failed-resolutions-bounded", "healthy-budget-refill"]
369+
const required = [
370+
"initialization-failure-retryable",
371+
"failed-resolutions-bounded",
372+
"healthy-budget-refill",
373+
"terminal-stop-after-retry-warning",
374+
]
346375
if (kind === "sidebar")
347376
required.push(
348377
"published-timeout-keeps-attempts",
@@ -407,13 +436,21 @@ function policyContracts(p: Policy): void {
407436
}
408437
for (let attempts = 0; attempts <= 3; attempts++)
409438
for (const reported of [false, true])
410-
for (const stop of [false, true]) {
411-
assert.deepEqual(
412-
p.recordRecoveryFailure({ attempts, reported }, stop),
413-
{ attempts: stop ? 3 : attempts, reported: true, notify: !reported },
414-
"failure accounting and notification deduplication",
415-
)
416-
}
439+
for (const stopLogged of [false, true])
440+
for (const stop of [false, true]) {
441+
assert.deepEqual(
442+
p.recordRecoveryFailure({ attempts, reported, stopLogged }, stop),
443+
{
444+
attempts: stop ? 3 : attempts,
445+
reported: true,
446+
notify: !reported,
447+
stopLogged: stopLogged || stop,
448+
log: stop ? !stopLogged : !reported && !stopLogged,
449+
},
450+
"failure accounting, notification deduplication, and terminal stop logging",
451+
)
452+
}
453+
417454
for (const started of [false, true])
418455
for (const focusSucceeded of [false, true])
419456
for (const pending of [false, true]) {
@@ -436,6 +473,28 @@ check(policy)
436473
// Representative policy regressions must be rejected, rather than merely
437474
// leaving a self-consistent abstract model green. These do not mutate source.
438475
const mutants: Array<[string, Policy, RegExp]> = [
476+
[
477+
"terminal stop suppressed after retry warning",
478+
{
479+
...policy,
480+
recordRecoveryFailure: (state, stop) => ({
481+
...policy.recordRecoveryFailure(state, stop),
482+
log: !state.reported,
483+
}),
484+
},
485+
/terminal stop logging/,
486+
],
487+
[
488+
"terminal stop repeated on every tick",
489+
{
490+
...policy,
491+
recordRecoveryFailure: (state, stop) => ({
492+
...policy.recordRecoveryFailure(state, stop),
493+
log: stop || !state.reported,
494+
}),
495+
},
496+
/terminal stop logging/,
497+
],
439498
[
440499
"stale notification reloads window",
441500
{ ...policy, recoveryWarningAction: (choice) => (choice === "Reload Window" ? "reload" : "none") },

‎src/core/webview/ClineProvider.ts‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,7 @@ export class ClineProvider
266266
// Keep that repair distinct from a user-opened view until it arrives.
267267
private pendingSidebarRecovery = false
268268
private webviewRecoveryFailureReported = false
269+
private webviewRecoveryStopLogged = false
269270
private lastReceivedWebviewHeartbeatAt: number | undefined
270271
private webviewHeartbeatCount = 0
271272
private webviewHealthySince: number | undefined
@@ -881,6 +882,7 @@ export class ClineProvider
881882
if (!preservesRecoveryIncident(this.webviewHostRecoveryInProgress, this.pendingSidebarRecovery)) {
882883
this.webviewRecoveryAttempts = 0
883884
this.webviewRecoveryFailureReported = false
885+
this.webviewRecoveryStopLogged = false
884886
}
885887
this.lastReceivedWebviewHeartbeatAt = undefined
886888
this.webviewHealthySince = undefined
@@ -3625,6 +3627,7 @@ export class ClineProvider
36253627
// A new heartbeat invalidates the old warning action. Allow a fresh warning
36263628
// if this renderer stalls again, without replenishing its retry budget.
36273629
this.webviewRecoveryFailureReported = false
3630+
this.webviewRecoveryStopLogged = false
36283631
if (this.webviewRecoveryConfirmation) {
36293632
this.log("[Zoo Code] Webview heartbeat returned after recovery")
36303633
}
@@ -3868,13 +3871,18 @@ export class ClineProvider
38683871
// A terminal failure still stops retries if an earlier retryable failure
38693872
// already showed the notification for this incident.
38703873
const failure = recordRecoveryFailure(
3871-
{ attempts: this.webviewRecoveryAttempts, reported: this.webviewRecoveryFailureReported },
3874+
{
3875+
attempts: this.webviewRecoveryAttempts,
3876+
reported: this.webviewRecoveryFailureReported,
3877+
stopLogged: this.webviewRecoveryStopLogged,
3878+
},
38723879
stopRecovery,
38733880
)
38743881
this.webviewRecoveryAttempts = failure.attempts
38753882
this.webviewRecoveryFailureReported = failure.reported
3883+
this.webviewRecoveryStopLogged = failure.stopLogged
3884+
if (failure.log) this.log(`[Zoo Code] Webview recovery ${stopRecovery ? "stopped" : "will retry"}: ${reason}`)
38763885
if (!failure.notify) return
3877-
this.log(`[Zoo Code] Webview recovery ${stopRecovery ? "stopped" : "will retry"}: ${reason}`)
38783886
const heartbeatCount = this.webviewHeartbeatCount
38793887
void Promise.resolve(
38803888
vscode.window.showWarningMessage(

‎src/core/webview/__tests__/ClineProvider.spec.ts‎

Lines changed: 61 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1668,20 +1668,59 @@ describe("ClineProvider", () => {
16681668
expect(provider.getCurrentTask()).toBeUndefined()
16691669
})
16701670

1671-
test("stops retries after a terminal failure even when a retry warning was already shown", async () => {
1672-
await provider.resolveWebviewView(mockWebviewView)
1673-
const html = mockWebviewView.webview.html
1674-
provider["reportWebviewRecoveryFailure"]("Initialization failed; retry pending", false)
1675-
expect(provider["webviewRecoveryAttempts"]).toBe(0)
1676-
expect(vscode.window.showWarningMessage).toHaveBeenCalledTimes(1)
1671+
test.each([false, true])(
1672+
"logs one terminal stop and notification despite repeated ticks (prior retry warning: %s)",
1673+
async (priorWarning) => {
1674+
await provider.resolveWebviewView(mockWebviewView)
1675+
const html = mockWebviewView.webview.html
1676+
const log = vi.spyOn(provider, "log")
1677+
if (priorWarning) {
1678+
provider["reportWebviewRecoveryFailure"]("Initialization failed; retry pending", false)
1679+
expect(provider["webviewRecoveryAttempts"]).toBe(0)
1680+
expect(log).toHaveBeenCalledWith(
1681+
"[Zoo Code] Webview recovery will retry: Initialization failed; retry pending",
1682+
)
1683+
}
16771684

1678-
provider["reportWebviewRecoveryFailure"]("Recovery is unavailable")
1679-
await vi.advanceTimersByTimeAsync(120_000)
1685+
provider["reportWebviewRecoveryFailure"]("Recovery is unavailable")
1686+
expect(log).toHaveBeenCalledWith("[Zoo Code] Webview recovery stopped: Recovery is unavailable")
1687+
await vi.advanceTimersByTimeAsync(300_000)
16801688

1681-
expect(provider["webviewRecoveryAttempts"]).toBe(3)
1682-
expect(mockWebviewView.webview.html).toBe(html)
1683-
expect(vscode.window.showWarningMessage).toHaveBeenCalledTimes(1)
1684-
})
1689+
expect(provider["webviewRecoveryAttempts"]).toBe(3)
1690+
expect(mockWebviewView.webview.html).toBe(html)
1691+
expect(log.mock.calls.filter(([line]) => line.includes("Webview recovery stopped:"))).toEqual([
1692+
["[Zoo Code] Webview recovery stopped: Recovery is unavailable"],
1693+
])
1694+
expect(vscode.window.showWarningMessage).toHaveBeenCalledTimes(1)
1695+
},
1696+
)
1697+
1698+
test.each(["heartbeat", "manual reopen"] as const)(
1699+
"logs a new terminal stop after %s begins a new incident",
1700+
async (reset) => {
1701+
await provider.resolveWebviewView(mockWebviewView)
1702+
const log = vi.spyOn(provider, "log")
1703+
provider["reportWebviewRecoveryFailure"]("First incident")
1704+
if (reset === "heartbeat") provider.updateWebviewHeartbeat()
1705+
else {
1706+
disposeCallback()
1707+
await provider.resolveWebviewView({
1708+
...mockWebviewView,
1709+
webview: { ...mockWebviewView.webview, html: "" },
1710+
})
1711+
}
1712+
expect(provider["webviewRecoveryStopLogged"]).toBe(false)
1713+
1714+
provider["reportWebviewRecoveryFailure"]("Second incident")
1715+
await vi.advanceTimersByTimeAsync(300_000)
1716+
1717+
expect(log.mock.calls.filter(([line]) => line.includes("Webview recovery stopped:"))).toEqual([
1718+
["[Zoo Code] Webview recovery stopped: First incident"],
1719+
["[Zoo Code] Webview recovery stopped: Second incident"],
1720+
])
1721+
expect(vscode.window.showWarningMessage).toHaveBeenCalledTimes(2)
1722+
},
1723+
)
16851724

16861725
test.each(["Reload Window", "Show Logs"] as const)(
16871726
"runs only the selected recovery action: %s",
@@ -2651,6 +2690,16 @@ describe("ClineProvider", () => {
26512690
expect(provider["webviewRecoveryAttempts"]).toBe(3)
26522691
expect(provider["webviewHostRecoveryInProgress"]).toBe(false)
26532692
expect(vscode.window.showWarningMessage).toHaveBeenCalledTimes(1)
2693+
expect(
2694+
vi
2695+
.mocked(mockOutputChannel.appendLine)
2696+
.mock.calls.filter(([line]) => line.includes("Webview recovery stopped:")),
2697+
).toHaveLength(1)
2698+
expect(mockOutputChannel.appendLine).toHaveBeenCalledWith(
2699+
expect.stringContaining(
2700+
"Webview recovery stopped: Recovery attempt limit reached without sustained heartbeats",
2701+
),
2702+
)
26542703
expect(provider["_disposed"]).toBe(false)
26552704
expect(provider.getCurrentTask()).toBe(task)
26562705
expect(task.dispose).not.toHaveBeenCalled()

‎src/core/webview/webviewRecoveryPolicy.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,13 +42,15 @@ export function pendingSidebarAfterCommandFailure(input: {
4242
}
4343

4444
export function recordRecoveryFailure(
45-
state: { attempts: number; reported: boolean },
45+
state: { attempts: number; reported: boolean; stopLogged: boolean },
4646
stopRecovery: boolean,
47-
): { attempts: number; reported: boolean; notify: boolean } {
47+
): { attempts: number; reported: boolean; notify: boolean; stopLogged: boolean; log: boolean } {
4848
return {
4949
attempts: stopRecovery ? WEBVIEW_RECOVERY_LIMITS.attempts : state.attempts,
5050
reported: true,
5151
notify: !state.reported,
52+
stopLogged: state.stopLogged || stopRecovery,
53+
log: stopRecovery ? !state.stopLogged : !state.reported && !state.stopLogged,
5254
}
5355
}
5456

0 commit comments

Comments
 (0)