Skip to content

Commit ec2b2b5

Browse files
authored
fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing (#3172)
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing A single drawElement frame could kill an entire render. On one comp (caption-editorial-emphasis) drawElementImage returns normally and the renderer then stops draining its task queue: the setTimeout(…, 0) that schedules toDataURL never fires, the capture page.evaluate never settles, and 60s later the stage watchdog fails the whole render. Deterministic on 152.0.7977.30, always the same frame. Root-cause detail in PRINFRA-488. Two gaps, both closed here. 1. Nothing below the 60s stage watchdog bounded a frame. Adds a per-frame deadline (HF_DE_FRAME_TIMEOUT_MS, default 15000, 0 disables) around the WHOLE frame operation, not just the drawElementImage call — the stall surfaces at whichever page round-trip comes next, and was observed at both the capture and the seek's background-image decode. Bounding one call missed it. Deliberately NO per-frame screenshot fallback for this error: once the renderer stops scheduling it is wedged for every subsequent round-trip on that page. Measured — the screenshot fallback blew the same deadline. The frame fails fast instead, so the recovery that actually works can run. 2. That recovery was ineligible. shouldRetryViaPinnedFallback only retried a generic capture failure when the worker count was PINNED by the inversion or the router; a comp that engaged drawElement on the ordinary single-worker path had no whole-render fallback at all. A renderer stall is now retryable on any routing — the failure is a property of drawElement itself, and the retry re-renders on a fresh page via screenshot. Reported as de_fallback_reason "de_renderer_stall" so it is distinguishable from capture_error in telemetry, and counted per-session as deFrameTimeouts. Verified end to end on the repro: previously exit 1 with "stalled: no frame progress for 60000ms"; now the deadline fires, the render retries via screenshot, and RENDER_OK with a valid 1920x1080 / 240-frame / 8.0s MP4. Tests cover the predicate both ways plus the cross-package error match. Engine 1,481 and producer 587 green (audioPadTrim.integration flakes only under full-suite parallel load — passes in isolation, and on clean HEAD it is green too, so it is not from this change). Refs PRINFRA-488 * fix(engine,producer,cli): count drawElement frame timeouts where they happen `session.deFrameTimeouts` was incremented inside `captureFrameCore`'s catch, but the deadline races that function from OUTSIDE it — `withFrameDeadline` is applied by `captureFrameToBuffer`, and `DeFrameTimeoutError` is constructed nowhere else. So the branch was unreachable: on a wedged renderer the counter stayed 0, the engine-side stall log never printed, and `CapturePerfSummary.deFrameTimeouts` reported 0 on every affected render. The end-to-end recovery worked throughout — `isDeRendererStallError` matches by name at the producer — only the observability was dead. Moved the increment and the log into an `onTimeout` hook that fires from the deadline itself, deleted the unreachable branch and the now-unused `isDeFrameTimeoutError`, and threaded the count out to `render_completed.de_frame_timeouts` so the rate is graphable apart from the other `capture_error` fallbacks. A stall always costs a whole-render re-run, which is why it deserves its own series. Also corrected the `fallbackReason` docstring, which still listed only `psnr | blank | oom | capture_error` after this branch added `de_renderer_stall`. `withFrameDeadline` is exported for a two-case fake-timer test: the deadline fires and calls `onTimeout` once, and a resolving inner promise passes its value through with no late timer. The first case fails if the counter moves back inside the work promise.
1 parent 564caa5 commit ec2b2b5

8 files changed

Lines changed: 244 additions & 4 deletions

File tree

‎packages/cli/src/commands/render.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1514,6 +1514,7 @@ function trackRenderMetrics(
15141514
deBlankRecaptures: perf?.drawElement?.blankRecaptures,
15151515
deBoundaryFrames: perf?.drawElement?.boundaryFrames,
15161516
deNcprFallbacks: perf?.drawElement?.ncprFallbacks,
1517+
deFrameTimeouts: perf?.drawElement?.frameTimeouts,
15171518
compositionDurationMs,
15181519
compositionWidth: perf?.resolution.width,
15191520
compositionHeight: perf?.resolution.height,

‎packages/cli/src/telemetry/events.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,7 @@ export function trackRenderComplete(
235235
deBlankRecaptures?: number;
236236
deBoundaryFrames?: number;
237237
deNcprFallbacks?: number;
238+
deFrameTimeouts?: number;
238239
// "cli" when triggered by `hyperframes render` (default), "studio" when
239240
// triggered by a studio preview-server render (POST /api/projects/:id/render).
240241
source?: "cli" | "studio";
@@ -336,6 +337,7 @@ export function trackRenderComplete(
336337
de_blank_recaptures: props.deBlankRecaptures,
337338
de_boundary_frames: props.deBoundaryFrames,
338339
de_ncpr_fallbacks: props.deNcprFallbacks,
340+
de_frame_timeouts: props.deFrameTimeouts,
339341
...powerStateFields(),
340342
source: props.source ?? "cli",
341343
composition_duration_ms: props.compositionDurationMs,
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
/**
2+
* Tests for the per-frame drawElement deadline (`withFrameDeadline`, PRINFRA-488).
3+
*
4+
* The deadline races the capture round-trip from OUTSIDE `captureFrameCore`,
5+
* because puppeteer cannot abort an in-flight `page.evaluate`. That is exactly
6+
* why the stall counter has to live in the `onTimeout` hook: a wedged renderer
7+
* never returns, so no catch block inside the work promise ever runs. The first
8+
* shipped version incremented `session.deFrameTimeouts` in that unreachable
9+
* catch, so the counter — and the `CapturePerfSummary` field it feeds — read 0
10+
* on every stalled render.
11+
*/
12+
13+
import { describe, expect, it, vi } from "vitest";
14+
import { withFrameDeadline } from "./frameCapture.js";
15+
16+
describe("withFrameDeadline", () => {
17+
it("rejects with DeFrameTimeoutError and fires onTimeout when work outlives the deadline", async () => {
18+
vi.useFakeTimers();
19+
try {
20+
const onTimeout = vi.fn();
21+
// Never settles — the wedged-renderer shape.
22+
const raced = withFrameDeadline(new Promise<string>(() => {}), "frame 7", 15_000, onTimeout);
23+
const assertion = expect(raced).rejects.toThrow(/frame 7 exceeded 15000ms/);
24+
await vi.advanceTimersByTimeAsync(15_000);
25+
await assertion;
26+
expect(onTimeout).toHaveBeenCalledTimes(1);
27+
} finally {
28+
vi.useRealTimers();
29+
}
30+
});
31+
32+
it("passes the value through and leaves onTimeout alone when work wins", async () => {
33+
vi.useFakeTimers();
34+
try {
35+
const onTimeout = vi.fn();
36+
const raced = withFrameDeadline(Promise.resolve("buffer"), "frame 7", 15_000, onTimeout);
37+
await expect(raced).resolves.toBe("buffer");
38+
// Past the deadline: the cleared timer must not fire late.
39+
await vi.advanceTimersByTimeAsync(30_000);
40+
expect(onTimeout).not.toHaveBeenCalled();
41+
} finally {
42+
vi.useRealTimers();
43+
}
44+
});
45+
});

‎packages/engine/src/services/frameCapture.ts‎

Lines changed: 94 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,13 @@ export interface CaptureSession {
212212
deVerifyInitMs?: number;
213213
/** Count of per-frame "No cached paint record" screenshot fallbacks (telemetry). */
214214
deNcprFallbacks?: number;
215+
/**
216+
* Count of drawElement frame captures that blew `HF_DE_FRAME_TIMEOUT_MS`
217+
* because the renderer stopped scheduling after drawElementImage returned
218+
* (PRINFRA-488). Each one aborts the drawElement attempt so the whole render
219+
* retries via screenshot.
220+
*/
221+
deFrameTimeouts?: number;
215222
/**
216223
* drawElement init passed every gate but stopped before verification +
217224
* canvas injection: the session has no video-frame injector yet (probe
@@ -3133,6 +3140,68 @@ function isNoCachedPaintRecordError(err: unknown): boolean {
31333140
return msg.includes("No cached paint record");
31343141
}
31353142

3143+
/**
3144+
* Per-frame deadline for the drawElement capture round-trip.
3145+
*
3146+
* drawElementImage can return normally and then leave the renderer not draining
3147+
* its task queue: the `setTimeout(…, 0)` that drawAndEncode schedules to run
3148+
* `toDataURL` never fires, so the capture `page.evaluate` never settles.
3149+
* Reproduced deterministically on Chromium 152.0.7977.30, one comp, always the
3150+
* same frame (PRINFRA-488). Nothing below the render-level watchdog bounded
3151+
* this, so a single bad frame failed the ENTIRE render after a 60 s stall.
3152+
*
3153+
* This bounds the round-trip so the frame can take the same per-frame screenshot
3154+
* fallback the `No cached paint record` case already takes — one slow frame
3155+
* instead of a dead render. Tune with `HF_DE_FRAME_TIMEOUT_MS`; 0 disables.
3156+
*/
3157+
const DE_FRAME_TIMEOUT_MS = Number(process.env.HF_DE_FRAME_TIMEOUT_MS ?? "15000");
3158+
3159+
class DeFrameTimeoutError extends Error {
3160+
constructor(label: string, ms: number) {
3161+
super(`drawElement ${label} exceeded ${ms}ms (renderer stopped scheduling; see PRINFRA-488)`);
3162+
this.name = "DeFrameTimeoutError";
3163+
}
3164+
}
3165+
3166+
/**
3167+
* Race `work` against a deadline. The losing promise is NOT cancellable —
3168+
* puppeteer cannot abort an in-flight `page.evaluate` — so its rejection is
3169+
* swallowed to avoid an unhandled rejection when it eventually settles (or
3170+
* never does). The orphaned round-trip keeps running in its Chrome worker;
3171+
* that worker is reclaimed by the outer retry rebuilding the page
3172+
* (`closeOrphanedProbeForRetry`), not by anything here.
3173+
*
3174+
* `onTimeout` fires exactly when the deadline wins, and is the ONLY place the
3175+
* stall is observable: because the deadline races `work` from outside, nothing
3176+
* inside `work` — including its own catch blocks — ever sees this error.
3177+
*
3178+
* Exported for the deadline unit test; `captureFrameToBuffer` is the only
3179+
* production caller.
3180+
*/
3181+
export async function withFrameDeadline<T>(
3182+
work: Promise<T>,
3183+
label: string,
3184+
ms: number,
3185+
onTimeout?: () => void,
3186+
): Promise<T> {
3187+
if (!(ms > 0)) return work;
3188+
let timer: ReturnType<typeof setTimeout> | undefined;
3189+
const guard = new Promise<never>((_resolve, reject) => {
3190+
timer = setTimeout(() => {
3191+
onTimeout?.();
3192+
reject(new DeFrameTimeoutError(label, ms));
3193+
}, ms);
3194+
});
3195+
try {
3196+
return await Promise.race([work, guard]);
3197+
} finally {
3198+
if (timer) clearTimeout(timer);
3199+
void work.catch(() => {
3200+
/* orphaned round-trip — see doc above */
3201+
});
3202+
}
3203+
}
3204+
31363205
async function captureFrameCore(
31373206
session: CaptureSession,
31383207
frameIndex: number,
@@ -3344,7 +3413,30 @@ export async function captureFrameToBuffer(
33443413
frameIndex: number,
33453414
time: number,
33463415
): Promise<CaptureBufferResult> {
3347-
const { buffer, captureTimeMs } = await captureFrameCore(session, frameIndex, time);
3416+
const { buffer, captureTimeMs } =
3417+
session.captureMode === "drawelement"
3418+
? await withFrameDeadline(
3419+
captureFrameCore(session, frameIndex, time),
3420+
`frame ${frameIndex}`,
3421+
DE_FRAME_TIMEOUT_MS,
3422+
() => {
3423+
// Deliberately NO per-frame screenshot fallback. When the renderer
3424+
// stops scheduling it is wedged for EVERY subsequent round-trip on
3425+
// that page — measured: the screenshot fallback blew the same
3426+
// deadline. Fail fast and let the producer re-render the whole comp
3427+
// on a fresh page via the screenshot path, the only recovery that
3428+
// works. Counted here rather than in captureFrameCore's catch: the
3429+
// deadline rejects from outside it, so that catch never runs.
3430+
session.deFrameTimeouts = (session.deFrameTimeouts ?? 0) + 1;
3431+
console.log(
3432+
`[engine] fast capture: frame ${frameIndex} — capture exceeded ` +
3433+
`${DE_FRAME_TIMEOUT_MS}ms; renderer stalled after drawElementImage ` +
3434+
`(PRINFRA-488). Failing the drawElement attempt so the whole render ` +
3435+
`retries via screenshot.`,
3436+
);
3437+
},
3438+
)
3439+
: await captureFrameCore(session, frameIndex, time);
33483440

33493441
return { buffer, captureTimeMs };
33503442
}
@@ -3951,5 +4043,6 @@ export function getCapturePerfSummary(session: CaptureSession): CapturePerfSumma
39514043
deVerifyInitMs: session.deVerifyInitMs ?? 0,
39524044
deBoundaryFrames: session.clipBoundaryFrames?.size ?? 0,
39534045
deNcprFallbacks: session.deNcprFallbacks ?? 0,
4046+
deFrameTimeouts: session.deFrameTimeouts ?? 0,
39544047
};
39554048
}

‎packages/engine/src/types.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,14 @@ export interface CapturePerfSummary {
345345
deBoundaryFrames: number;
346346
/** Per-frame "No cached paint record" screenshot fallbacks during capture. */
347347
deNcprFallbacks: number;
348+
/**
349+
* Per-frame drawElement captures that blew the `HF_DE_FRAME_TIMEOUT_MS`
350+
* deadline and took the screenshot fallback (renderer stopped scheduling
351+
* after drawElementImage returned — PRINFRA-488). Non-zero means the render
352+
* completed only because the deadline caught a stall that previously failed
353+
* the whole render.
354+
*/
355+
deFrameTimeouts: number;
348356
}
349357

350358
// ── Global Augmentation ────────────────────────────────────────────────────────

‎packages/producer/src/services/render/perfSummary.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@ function aggregateDrawElement(
149149
blankRecaptures: drain?.blankRecaptures ?? 0,
150150
boundaryFrames: perfs.reduce((sum, p) => sum + (p.deBoundaryFrames ?? 0), 0),
151151
ncprFallbacks: perfs.reduce((sum, p) => sum + (p.deNcprFallbacks ?? 0), 0),
152+
frameTimeouts: perfs.reduce((sum, p) => sum + (p.deFrameTimeouts ?? 0), 0),
152153
};
153154
}
154155

‎packages/producer/src/services/renderOrchestrator.test.ts‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ import {
3434
resolveParallelRouterRetryPlan,
3535
resetCaptureAttemptProgress,
3636
shouldRetryViaPinnedFallback,
37+
isDeRendererStallError,
3738
countElementTags,
3839
envInt,
3940
isDeParallelRouterEnabled,
@@ -2457,6 +2458,60 @@ describe("resolveParallelRouterRetryPlan (self-verify retry rollback)", () => {
24572458
});
24582459

24592460
describe("shouldRetryViaPinnedFallback (widen the self-verify retry to generic capture failures, including OOM)", () => {
2461+
// PRINFRA-488: a wedged renderer must be retryable on ANY routing. Before this,
2462+
// a comp that engaged drawElement on the ordinary single-worker path had no
2463+
// whole-render fallback, so one stalled frame failed the entire render.
2464+
it("retries a drawElement renderer stall even with no pinned routing", () => {
2465+
expect(
2466+
shouldRetryViaPinnedFallback({
2467+
isVerifyError: false,
2468+
isCancellation: false,
2469+
deWorkerInversion: undefined,
2470+
deParallelRouter: undefined,
2471+
isDeRendererStall: true,
2472+
}),
2473+
).toBe(true);
2474+
});
2475+
2476+
it("still does NOT retry a generic capture failure with no pinned routing", () => {
2477+
expect(
2478+
shouldRetryViaPinnedFallback({
2479+
isVerifyError: false,
2480+
isCancellation: false,
2481+
deWorkerInversion: undefined,
2482+
deParallelRouter: undefined,
2483+
isDeRendererStall: false,
2484+
}),
2485+
).toBe(false);
2486+
});
2487+
2488+
it("never retries a cancellation, even for a renderer stall", () => {
2489+
expect(
2490+
shouldRetryViaPinnedFallback({
2491+
isVerifyError: false,
2492+
isCancellation: true,
2493+
deWorkerInversion: undefined,
2494+
deParallelRouter: undefined,
2495+
isDeRendererStall: true,
2496+
}),
2497+
).toBe(false);
2498+
});
2499+
2500+
it("recognizes the engine's stall error across the package boundary", () => {
2501+
const byName = new Error("whatever");
2502+
byName.name = "DeFrameTimeoutError";
2503+
expect(isDeRendererStallError(byName)).toBe(true);
2504+
expect(
2505+
isDeRendererStallError(
2506+
new Error(
2507+
"drawElement frame 50 exceeded 15000ms (renderer stopped scheduling; see PRINFRA-488)",
2508+
),
2509+
),
2510+
).toBe(true);
2511+
expect(isDeRendererStallError(new Error("some other capture failure"))).toBe(false);
2512+
expect(isDeRendererStallError("not an error")).toBe(false);
2513+
});
2514+
24602515
it("always retries a drawElement self-verify failure, pinned or not", () => {
24612516
expect(
24622517
shouldRetryViaPinnedFallback({

‎packages/producer/src/services/renderOrchestrator.ts‎

Lines changed: 38 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -529,9 +529,9 @@ export interface RenderPerfSummary {
529529
* `fallbackReason` being set is the "any fallback fired" signal.
530530
*/
531531
selfVerifyFallback: boolean;
532-
/** What tripped the fallback retry: psnr | blank | oom | capture_error. */
532+
/** What tripped the fallback retry: psnr | blank | oom | de_renderer_stall | capture_error. */
533533
fallbackReason?: string;
534-
/** The failing PSNR (dB) when `fallbackReason === "psnr"`; undefined for blank/oom/capture_error (no score exists). */
534+
/** The failing PSNR (dB) when `fallbackReason === "psnr"`; undefined for every other reason (no score exists). */
535535
fallbackFailedDb?: number;
536536
/** Frame index the verification failure was detected at; set for both "psnr" and "blank" fallback reasons. */
537537
fallbackFrameIndex?: number;
@@ -545,6 +545,13 @@ export interface RenderPerfSummary {
545545
boundaryFrames: number;
546546
/** Per-frame "No cached paint record" screenshot fallbacks. */
547547
ncprFallbacks: number;
548+
/**
549+
* Frames that blew `HF_DE_FRAME_TIMEOUT_MS` — a wedged renderer
550+
* (PRINFRA-488). Distinct from the other fallback counters: this one always
551+
* costs a whole-render re-run via screenshot, so its rate is worth graphing
552+
* on its own rather than inside `capture_error`.
553+
*/
554+
frameTimeouts: number;
548555
};
549556
}
550557

@@ -1808,12 +1815,34 @@ export function shouldRetryViaPinnedFallback(args: {
18081815
isCancellation: boolean;
18091816
deWorkerInversion: "inverted" | "reverted" | undefined;
18101817
deParallelRouter: "routed" | "reverted" | undefined;
1818+
/**
1819+
* The drawElement capture wedged the renderer (PRINFRA-488). Retryable on ANY
1820+
* routing, not just a pinned one: the failure is a property of drawElement
1821+
* itself, and the retry re-renders on a fresh page via screenshot — the only
1822+
* recovery that works once the renderer stops scheduling. Without this a comp
1823+
* that engaged drawElement on the ordinary single-worker path (neither
1824+
* inverted nor routed) had NO whole-render fallback, so one wedged frame
1825+
* failed the entire render.
1826+
*/
1827+
isDeRendererStall?: boolean;
18111828
}): boolean {
18121829
if (args.isCancellation) return false;
18131830
if (args.isVerifyError) return true;
1831+
if (args.isDeRendererStall === true) return true;
18141832
return args.deWorkerInversion === "inverted" || args.deParallelRouter === "routed";
18151833
}
18161834

1835+
/**
1836+
* True for the drawElement per-frame deadline breach raised by the engine when
1837+
* the renderer stops scheduling after `drawElementImage` returns (PRINFRA-488).
1838+
* Matched on name+message rather than by class because the error crosses the
1839+
* engine/producer package boundary.
1840+
*/
1841+
export function isDeRendererStallError(err: unknown): boolean {
1842+
if (!(err instanceof Error)) return false;
1843+
return err.name === "DeFrameTimeoutError" || err.message.includes("renderer stopped scheduling");
1844+
}
1845+
18171846
/**
18181847
* When a self-verify (or pinned-fallback) retry is triggered mid-capture, the
18191848
* caller may still hold a live probe session that the failed stage was passed
@@ -3556,6 +3585,7 @@ async function executeRenderPipeline(input: {
35563585
// spawns on retry. See shouldRetryViaPinnedFallback for exactly
35573586
// which errors qualify.
35583587
const isVerifyError = isDrawElementVerificationError(err);
3588+
const isDeStall = isDeRendererStallError(err);
35593589
const isCancellation =
35603590
err instanceof RenderCancelledError || executionSignal?.aborted === true;
35613591
if (
@@ -3564,6 +3594,7 @@ async function executeRenderPipeline(input: {
35643594
isCancellation,
35653595
deWorkerInversion,
35663596
deParallelRouter,
3597+
isDeRendererStall: isDeStall,
35673598
})
35683599
)
35693600
throw err;
@@ -3576,7 +3607,11 @@ async function executeRenderPipeline(input: {
35763607
deFallbackFrameIndex = t.frameIndex;
35773608
deFallbackThresholdDb = t.thresholdDb;
35783609
} else {
3579-
deFallbackReason = isMemoryExhaustion ? "oom" : "capture_error";
3610+
deFallbackReason = isMemoryExhaustion
3611+
? "oom"
3612+
: isDeStall
3613+
? "de_renderer_stall"
3614+
: "capture_error";
35803615
}
35813616
log.warn(
35823617
isVerifyError

0 commit comments

Comments
 (0)