Skip to content

Commit 3202f3f

Browse files
miga-heygenclaude
andauthored
fix(producer): trip DE parallel-router circuit breaker on stalls and hangs (#3479)
Root cause: the per-worker capture calls in captureFrameRange (parallelCoordinator.ts) take no abort signal of their own, and only checked `signal.aborted` BEFORE starting each frame — a no-op once a worker is already awaiting an in-flight call. On WSL2, the native drawElement/BeginFrame capture call can hang indefinitely at frame 0 with no error. The DE parallel-router's existing stall watchdog (captureStreamingStage.ts) correctly fires `stallController.abort()` after HF_DE_STALL_MS, but that abort had no way to reach a worker already wedged inside a hung capture call — so executeParallelCapture's Promise.all waited forever, the render hung indefinitely, and the CLI's circuit breaker (which only runs after executeRenderJob settles) never got a chance to trip. Fix: race each per-frame capture call against the signal actually firing (raceAgainstAbort), the same "can't cancel, only race" pattern already used by the sequential capture path. Once the watchdog's abort is observed, the wedged worker rejects, executeParallelCapture settles, and the existing pinned-fallback retry / "reverted" outcome / circuit breaker machinery (already correct) runs end to end. Also widen the CLI breaker's trip condition from the literal string "reverted" to "not a clean routed success", so any future non-success outcome the observability layer records also latches the breaker instead of silently falling through. Closes #3441 Co-authored-by: Miga <[email protected]>
1 parent 6eaa2cb commit 3202f3f

3 files changed

Lines changed: 115 additions & 10 deletions

File tree

‎packages/cli/src/commands/render.ts‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1314,8 +1314,9 @@ function persistDeParallelRouterTrialFired(): boolean {
13141314

13151315
/**
13161316
* After a trial-armed render, persist that the router's OWN bet actually
1317-
* failed — its self-verify/generic-failure safety net fired
1318-
* (`deParallelRouter === "reverted"`) — or that the render-count backstop
1317+
* failed — its self-verify/generic-failure safety net fired (recorded as
1318+
* anything other than a clean `"routed"`, e.g. `"reverted"`, or a stall/hang
1319+
* outcome — heygen-com/hyperframes#3441) — or that the render-count backstop
13191320
* (`DE_PARALLEL_ROUTER_TRIAL_MAX_RENDERS`) was reached, so it's never
13201321
* enabled again for this install. A clean "routed" (the render succeeded
13211322
* with no fallback) does NOT consume the trial by itself — the whole point
@@ -1352,11 +1353,19 @@ function maybeConsumeDeParallelRouterTrial(
13521353
const config = readConfigFresh();
13531354
const renderCount = (config.deParallelRouterTrialRenderCount ?? 0) + 1;
13541355
config.deParallelRouterTrialRenderCount = renderCount;
1355-
// Trip ONLY on an actual fallback. The old trial also tripped at a
1356-
// 25-render exposure cap, which was sampling logic: bound how long an
1357-
// experiment force-enables itself. Under a shipped default that would
1358-
// switch the feature off behind the user's back after 25 good renders.
1359-
const fired = outcome === "reverted";
1356+
// Trip on any recorded non-success, not only the literal string
1357+
// "reverted". The old trial also tripped at a 25-render exposure cap,
1358+
// which was sampling logic: bound how long an experiment force-enables
1359+
// itself. Under a shipped default that would switch the feature off
1360+
// behind the user's back after 25 good renders — so a clean "routed" (no
1361+
// fallback needed) must NOT trip. But narrowing the positive check to the
1362+
// single string "reverted" (heygen-com/hyperframes#3441) meant any other
1363+
// non-success signal the observability layer might ever record — a stall,
1364+
// a timeout, a future outcome value — would silently fall through to "not
1365+
// fired" instead of tripping. `outcome` is `undefined`-filtered above, so
1366+
// by this point it is a real recorded outcome; the only one that means
1367+
// "no fallback happened" is "routed" itself.
1368+
const fired = outcome !== "routed";
13601369
if (fired) {
13611370
config.deParallelRouterTrialFired = true;
13621371
// Latch BEFORE attempting persistence — the decision holds for this

‎packages/engine/src/services/parallelCoordinator-peerAbort.test.ts‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,4 +61,57 @@ describe("executeParallelCapture peer abort", () => {
6161
rmSync(root, { recursive: true, force: true });
6262
}
6363
});
64+
65+
// heygen-com/hyperframes#3441: a worker wedged INSIDE a native capture call
66+
// (WSL2 hangs the very first drawElement/BeginFrame call at frame 0 with no
67+
// error) must actually be unstuck once the caller's `signal` aborts — e.g.
68+
// the DE parallel-router stall watchdog in `captureStreamingStage.ts` firing
69+
// after HF_DE_STALL_MS. Before this fix, `captureFrameRange` only checked
70+
// `signal.aborted` BEFORE starting each frame's capture call, which is a
71+
// no-op for a call that is already in flight and never settles — the abort
72+
// had no way to reach it, so `executeParallelCapture`'s `Promise.all` (and
73+
// therefore the whole render) hung forever, and the CLI circuit breaker
74+
// (which only runs after `executeRenderJob` settles) never got a chance to
75+
// trip.
76+
it("rejects promptly when the signal aborts while a worker is wedged inside a capture call that never settles", async () => {
77+
const root = mkdtempSync(join(tmpdir(), "hf-stall-abort-"));
78+
// Simulates the native capture call hanging indefinitely (never resolves,
79+
// never rejects) — exactly the WSL2 shape from the field report.
80+
const captureFrame = vi.fn(() => new Promise<void>(() => {}));
81+
const closeCaptureSession = vi.fn().mockResolvedValue(undefined);
82+
vi.doMock("./frameCapture.js", () => ({
83+
createCaptureSession: vi.fn(
84+
async () => ({ workerId: 0, browserConsoleBuffer: [] }) as unknown as CaptureSession,
85+
),
86+
initializeSession: vi.fn(async () => {}),
87+
captureFrame,
88+
captureFrameToBuffer: vi.fn(),
89+
captureFrameToBufferPipelined: vi.fn(),
90+
closeCaptureSession,
91+
getCapturePerfSummary: vi.fn(() => ({ frames: 0 })),
92+
}));
93+
94+
try {
95+
const { executeParallelCapture } = await import("./parallelCoordinator.js");
96+
const controller = new AbortController();
97+
const result = executeParallelCapture(
98+
"http://127.0.0.1",
99+
root,
100+
[{ workerId: 0, startFrame: 0, endFrame: 3, outputDir: join(root, "worker-0") }],
101+
{ width: 320, height: 180, fps: { num: 30, den: 1 } },
102+
() => null,
103+
controller.signal,
104+
);
105+
106+
// Give the worker a tick to actually enter the (never-resolving)
107+
// capture call before simulating the watchdog's abort.
108+
await new Promise((resolve) => setTimeout(resolve, 10));
109+
controller.abort();
110+
111+
await expect(result).rejects.toMatchObject({ name: "CaptureFailure" });
112+
expect(captureFrame).toHaveBeenCalledTimes(1);
113+
} finally {
114+
rmSync(root, { recursive: true, force: true });
115+
}
116+
});
64117
});

‎packages/engine/src/services/parallelCoordinator.ts‎

Lines changed: 46 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -461,6 +461,43 @@ export function shouldVerifyWorkerGpu(workerId: number, config?: Partial<EngineC
461461
return config?.browserGpuMode === "software" && workerId === 0;
462462
}
463463

464+
/**
465+
* Race a single in-flight capture call against `signal` actually firing.
466+
*
467+
* `captureFrame`/`captureFrameToBuffer`/`captureFrameToBufferPipelined` take
468+
* no abort signal of their own — a native browser call wedged inside one of
469+
* them (WSL2 hangs the very first drawElement/BeginFrame capture at frame 0
470+
* with no error, heygen-com/hyperframes#3441) cannot be cancelled, only
471+
* raced. Without this, the `signal?.aborted` checks at the top of the
472+
* `captureFrameRange` loop are a no-op the moment a worker is already
473+
* awaiting a hung call: nothing revisits that check until the await settles,
474+
* which on a genuine hang is never. The DE parallel-router's stall watchdog
475+
* (`captureStreamingStage.ts`) does fire `stallController.abort()` after
476+
* `HF_DE_STALL_MS`, but until this call actually observes the signal, that
477+
* abort has no effect on an already-wedged worker — `executeParallelCapture`'s
478+
* `Promise.all` waits forever, so the render hangs indefinitely and the CLI's
479+
* circuit breaker (which only runs after `executeRenderJob` settles) never
480+
* gets a chance to trip.
481+
*/
482+
function raceAgainstAbort<T>(promise: Promise<T>, signal: AbortSignal | undefined): Promise<T> {
483+
if (!signal) return promise;
484+
if (signal.aborted) return Promise.reject(new Error("Parallel worker cancelled"));
485+
return new Promise<T>((resolve, reject) => {
486+
const onAbort = () => reject(new Error("Parallel worker cancelled"));
487+
signal.addEventListener("abort", onAbort, { once: true });
488+
promise.then(
489+
(value) => {
490+
signal.removeEventListener("abort", onAbort);
491+
resolve(value);
492+
},
493+
(error) => {
494+
signal.removeEventListener("abort", onAbort);
495+
reject(error);
496+
},
497+
);
498+
});
499+
}
500+
464501
// fallow-ignore-next-line complexity
465502
async function captureFrameRange(
466503
session: CaptureSession,
@@ -498,7 +535,10 @@ async function captureFrameRange(
498535
if (dbg && i < task.startFrame + dbgWin) {
499536
console.log(`[par:w${task.workerId}] +${Date.now() - dbgT0}ms produce ${i} start`);
500537
}
501-
const { encodeResult } = await captureFrameToBufferPipelined(session, i - outputOffset, time);
538+
const { encodeResult } = await raceAgainstAbort(
539+
captureFrameToBufferPipelined(session, i - outputOffset, time),
540+
signal,
541+
);
502542
// Marks the promise "handled" for Node's unhandled-rejection detector
503543
// without affecting the real `await prev.encodeResult` below — if a
504544
// later iteration throws (abort, downstream writeFrame failure) before
@@ -542,10 +582,13 @@ async function captureFrameRange(
542582
const fileFrameIdx = i - outputOffset;
543583

544584
if (onFrameBuffer) {
545-
const { buffer } = await captureFrameToBuffer(session, fileFrameIdx, time);
585+
const { buffer } = await raceAgainstAbort(
586+
captureFrameToBuffer(session, fileFrameIdx, time),
587+
signal,
588+
);
546589
await onFrameBuffer(i, buffer, session);
547590
} else {
548-
await captureFrame(session, fileFrameIdx, time);
591+
await raceAgainstAbort(captureFrame(session, fileFrameIdx, time), signal);
549592
}
550593
framesCaptured++;
551594
if (onFrameCaptured) onFrameCaptured(task.workerId, i);

0 commit comments

Comments
 (0)