Skip to content

Commit bde9b88

Browse files
vanceingallsclaude
andcommitted
perf(producer): offload shader-blend to worker_threads to close JS event-loop ceiling
Follow-up to hf#732 (commit 4222971). The prior fix parallelized the DOM-session work for layered transition frames across N browser sessions, but the per-pixel JS shader-blend at the tail of `processLayeredTransitionFrame` (`TRANSITIONS[shader](from, to, output, w, h, progress)`) still executed on the Node main event loop. Complex shaders (`domain-warp`, `swirl-vortex`, `glitch`) iterate every pixel of the rgb48le buffer with multiple noise/sample calls per pixel — hundreds of milliseconds per call — so N concurrent DOM workers all firing shader-blends saturated the single thread. The worker-count sweep on the hf#677 fixture flattened after w=2 (w=1=218s, w=2=183s, w=6=184s, w=12=188s) — the classic single-threaded-downstream signature. This change moves the shader-blend onto a Node `worker_threads` pool: - `shaderTransitionWorkerPool.ts` — pool spawned once per layered render (sized to `min(layeredWorkerCount, cpuCount)`), terminated in the `finally`. Idle/busy slots with a FIFO queue; each task posts the three rgb48le `ArrayBuffer`s to a worker via `transferList` for zero-copy detach, runs the blend, posts the output back (also via `transferList`). Main-thread Buffer references are swapped to the re-attached views post-await. - `shaderTransitionWorker.ts` — worker entry. Imports `TRANSITIONS` and `crossfade` from a new `@hyperframes/engine/shader-transitions` subpath export that points straight at the standalone `shaderTransitions.ts` file. The subpath sidesteps a tsx / worker_threads loader-rewrite limitation: the engine's root index transitively imports `./config.js` and other files, and the `.js → .ts` rewrite does not survive the Worker boundary in dev/test. The standalone file has zero internal imports so the worker loads cleanly under both Node-direct (prod) and tsx/vitest (dev/test). - `renderOrchestrator.ts` — `processLayeredTransitionFrame` takes an optional `shaderPool` arg. When the pool is present (hybrid SDR multi-worker path with transitions), the shader-blend dispatches off the main thread; otherwise the legacy synchronous path runs (HDR fallback, single-worker, all-transition edge case). Both branches produce byte-identical output. A new `transitionShaderBlendMs` timing key surfaces the actual blend cost in `perf-summary.json`. - `build.mjs` — adds `shaderTransitionWorker.ts` as a third esbuild entry point so `dist/services/shaderTransitionWorker.js` is shipped for production loads. The pool probes for the `.js` first and falls back to the `.ts` source in dev. - `cli/tsup.config.ts` — mirrors the `@hyperframes/engine/shader-transitions` alias so the CLI's tsup bundle resolves the subpath the same way the producer's build does. Buffer transfer semantics: `transferList: [bufferA.buffer, bufferB.buffer, output.buffer]` detaches the original ArrayBuffers on the sender (Buffer `.length` collapses to 0). The worker posts the same three buffers back, also via transferList. The orchestrator re-points `buffers.bufferA` / `buffers.bufferB` / `buffers.output` at the returned Buffer views before the next iteration touches them. The mutation is documented in the `LayeredTransitionBuffers` docblock. Pool lifecycle is bounded by the hybrid dispatch block. If pool spawn fails (e.g. resource exhaustion) we log a warn and fall back to the inline blend path rather than failing the render. The pool's `terminate()` rejects both queued and in-flight tasks before `Worker.terminate()` to avoid leaks; mid-render worker crashes are treated as fatal (the in-flight buffers have been transferred and cannot be reconstructed). Tests: - `shaderTransitionWorkerPool.test.ts`: 6 unit tests covering byte-equivalence with the inline path across all 15 shaders, transferList detach semantics, unknown-shader fallback to crossfade, concurrent dispatch correctness across a 4-worker pool, and clean terminate-without-hanging on a busy pool. - Existing `renderOrchestrator.test.ts` regressions: 22 layered/hybrid tests still pass; the one pre-existing Linux/Windows path failure remains untouched. — Vai Co-Authored-By: Vai <[email protected]>
1 parent 4222971 commit bde9b88

7 files changed

Lines changed: 871 additions & 3 deletions

File tree

‎packages/cli/tsup.config.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,15 @@ var __dirname = __hf_dirname(__filename);`,
5656
esbuildOptions(options) {
5757
options.alias = {
5858
"@hyperframes/producer": resolve(__dirname, "../producer/src/index.ts"),
59+
// hf#677 follow-up: the producer's shader-blend worker imports from
60+
// `@hyperframes/engine/shader-transitions` (subpath export) for a
61+
// standalone TS file with zero internal imports — survives the
62+
// worker_thread loader boundary. Mirror the alias here so tsup's
63+
// bundler resolves it the same way as the producer's own build.mjs.
64+
"@hyperframes/engine/shader-transitions": resolve(
65+
__dirname,
66+
"../engine/src/utils/shaderTransitions.ts",
67+
),
5968
};
6069
options.loader = { ...options.loader, ".browser.js": "text" };
6170
},

‎packages/engine/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@
1111
"main": "./src/index.ts",
1212
"types": "./src/index.ts",
1313
"exports": {
14-
".": "./src/index.ts"
14+
".": "./src/index.ts",
15+
"./shader-transitions": "./src/utils/shaderTransitions.ts"
1516
},
1617
"scripts": {
1718
"build": "tsc",

‎packages/producer/build.mjs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ const workspaceAliasPlugin = {
2121
build.onResolve({ filter: /^@hyperframes\/engine$/ }, () => ({
2222
path: resolve(scriptDir, "../engine/src/index.ts"),
2323
}));
24+
build.onResolve({ filter: /^@hyperframes\/engine\/shader-transitions$/ }, () => ({
25+
path: resolve(scriptDir, "../engine/src/utils/shaderTransitions.ts"),
26+
}));
2427
build.onResolve({ filter: /^@hyperframes\/core$/ }, () => ({
2528
path: resolve(scriptDir, "../core/src/index.ts"),
2629
}));
@@ -55,6 +58,24 @@ await Promise.all([
5558
entryPoints: ["src/server.ts"],
5659
outfile: "dist/public-server.js",
5760
}),
61+
// Shader-blend worker (hf#677 follow-up). Loaded by
62+
// `shaderTransitionWorkerPool.createShaderTransitionWorkerPool` via
63+
// `new Worker(<path>)`. The pool probes for the `.js` build first,
64+
// falling back to the in-tree `.ts` source under tsx in dev. Must be
65+
// a separate entry point so it's bundled into a standalone worker
66+
// module — `new Worker(...)` cannot share the parent module graph.
67+
build({
68+
bundle: true,
69+
platform: "node",
70+
target: "node22",
71+
format: "esm",
72+
external: ["puppeteer", "esbuild", "postcss"],
73+
plugins: [workspaceAliasPlugin],
74+
minify: false,
75+
sourcemap: true,
76+
entryPoints: ["src/services/shaderTransitionWorker.ts"],
77+
outfile: "dist/services/shaderTransitionWorker.js",
78+
}),
5879
]);
5980

6081
// Copy core runtime artifacts so the producer can find them at dist/

‎packages/producer/src/services/renderOrchestrator.ts‎

Lines changed: 102 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,10 @@ import { randomUUID } from "crypto";
9898
import { freemem } from "os";
9999
import { fileURLToPath } from "url";
100100
import { createFileServer, type FileServerHandle, VIRTUAL_TIME_SHIM } from "./fileServer.js";
101+
import {
102+
createShaderTransitionWorkerPool,
103+
type ShaderTransitionWorkerPool,
104+
} from "./shaderTransitionWorkerPool.js";
101105
import { type CompiledComposition } from "./htmlCompiler.js";
102106
import { defaultLogger, type ProducerLogger } from "../logger.js";
103107
import { isPathInside } from "../utils/paths.js";
@@ -401,6 +405,7 @@ type HdrPerfTimingKey =
401405
| "canvasClearMs"
402406
| "normalCompositeMs"
403407
| "transitionCompositeMs"
408+
| "transitionShaderBlendMs"
404409
| "encoderWriteMs"
405410
| "hdrVideoReadDecodeMs"
406411
| "hdrVideoTransferMs"
@@ -440,6 +445,7 @@ function createHdrPerfCollector(): HdrPerfCollector {
440445
canvasClearMs: 0,
441446
normalCompositeMs: 0,
442447
transitionCompositeMs: 0,
448+
transitionShaderBlendMs: 0,
443449
encoderWriteMs: 0,
444450
hdrVideoReadDecodeMs: 0,
445451
hdrVideoTransferMs: 0,
@@ -481,6 +487,10 @@ function finalizeHdrPerf(perf: HdrPerfCollector): HdrPerfSummary {
481487
perf.timings.transitionCompositeMs,
482488
perf.transitionFrames,
483489
);
490+
avgMs.transitionShaderBlendMs = averageTiming(
491+
perf.timings.transitionShaderBlendMs,
492+
perf.transitionFrames,
493+
);
484494

485495
const perDomLayerKeys: HdrPerfTimingKey[] = [
486496
"domLayerSeekMs",
@@ -1835,6 +1845,11 @@ async function compositeHdrFrame(
18351845
// against their own browser concurrently.
18361846

18371847
interface LayeredTransitionBuffers {
1848+
// Mutable fields: the hf#677 shader-pool path swaps these references
1849+
// after each transferList round-trip with the worker thread. The
1850+
// underlying memory is the same; the Buffer headers are fresh views
1851+
// over the re-attached ArrayBuffers. The legacy synchronous path
1852+
// never reassigns these.
18381853
bufferA: Buffer;
18391854
bufferB: Buffer;
18401855
output: Buffer;
@@ -1934,6 +1949,16 @@ export async function processLayeredNormalFrame(
19341949
* @param nativeHdrIds - Set of HDR element ids; passed to mask helpers so HDR
19351950
* elements stay inline-hidden behind the screenshot
19361951
* (their pixels arrive via `blitHdrVideoLayer` etc.).
1952+
* @param shaderPool - Optional Node `worker_threads` pool to which the
1953+
* per-pixel shader-blend is dispatched (hf#677 follow-up
1954+
* to close the JS event-loop ceiling). When provided,
1955+
* bufferA/bufferB/output ArrayBuffers are detached via
1956+
* `transferList`, blended on a Worker, and transferred
1957+
* back; the function reassigns the fields on `buffers`
1958+
* to the returned Buffer views before returning. When
1959+
* omitted (e.g. legacy sequential path / single-worker
1960+
* SDR / HDR fallback), the blend runs synchronously on
1961+
* the calling thread — bit-for-bit identical output.
19371962
*/
19381963
export async function processLayeredTransitionFrame(
19391964
session: CaptureSession,
@@ -1945,6 +1970,7 @@ export async function processLayeredTransitionFrame(
19451970
buffers: LayeredTransitionBuffers,
19461971
assertNotAborted: () => void,
19471972
nativeHdrIds: Set<string>,
1973+
shaderPool?: ShaderTransitionWorkerPool | null,
19481974
): Promise<void> {
19491975
const {
19501976
log,
@@ -2077,8 +2103,41 @@ export async function processLayeredTransitionFrame(
20772103
}
20782104
}
20792105

2080-
const transitionFn: TransitionFn = TRANSITIONS[transition.shader] ?? crossfade;
2081-
transitionFn(buffers.bufferA, buffers.bufferB, buffers.output, width, height, progress);
2106+
// Shader-blend dispatch. When a `shaderPool` is provided (hf#677 follow-up
2107+
// for the JS event-loop ceiling), the per-pixel blend runs on a Node
2108+
// `worker_threads` Worker via zero-copy `transferList`. The pool
2109+
// detaches the bufferA/B/output ArrayBuffers, runs the blend, and
2110+
// transfers them back; we reattach the returned Buffers onto the
2111+
// caller's `buffers` slot so the next iteration of the worker's frame
2112+
// loop sees the same shape.
2113+
//
2114+
// Without a pool, the blend runs synchronously on the main thread — the
2115+
// legacy path. Both branches produce byte-identical output because the
2116+
// worker imports the same `TRANSITIONS` table from `@hyperframes/engine`.
2117+
if (shaderPool) {
2118+
const blendStart = Date.now();
2119+
const result = await shaderPool.run({
2120+
shader: transition.shader,
2121+
bufferA: buffers.bufferA,
2122+
bufferB: buffers.bufferB,
2123+
output: buffers.output,
2124+
width,
2125+
height,
2126+
progress,
2127+
});
2128+
// The originals are detached; swap in the transferred-back views so
2129+
// the caller's `LayeredTransitionBuffers` stay valid for the next
2130+
// frame. Same underlying memory, fresh Buffer headers.
2131+
buffers.bufferA = result.bufferA;
2132+
buffers.bufferB = result.bufferB;
2133+
buffers.output = result.output;
2134+
addHdrTiming(hdrPerf, "transitionShaderBlendMs", blendStart);
2135+
} else {
2136+
const blendStart = Date.now();
2137+
const transitionFn: TransitionFn = TRANSITIONS[transition.shader] ?? crossfade;
2138+
transitionFn(buffers.bufferA, buffers.bufferB, buffers.output, width, height, progress);
2139+
addHdrTiming(hdrPerf, "transitionShaderBlendMs", blendStart);
2140+
}
20822141
addHdrTiming(hdrPerf, "transitionCompositeMs", transitionTimingStart);
20832142
}
20842143

@@ -3464,6 +3523,20 @@ export async function executeRenderJob(
34643523
if (hybridEligible) {
34653524
const workerSessions: CaptureSession[] = [];
34663525
const workerCanvasesNeeded = Math.max(0, layeredWorkerCount - 1);
3526+
3527+
// hf#677 follow-up: spawn a worker_threads pool for the per-pixel
3528+
// shader-blend. The prior hf#732 commit parallelized DOM-session
3529+
// work across `layeredWorkerCount` browsers, but the JS shader
3530+
// call at the tail of `processLayeredTransitionFrame` still
3531+
// executed on the Node main event loop — six DOM workers all
3532+
// firing `TRANSITIONS[shader](...)` saturated the single thread
3533+
// and the worker-count sweep flattened after w=2. Moving the
3534+
// blend onto `worker_threads` removes that ceiling. Pool size
3535+
// matches `layeredWorkerCount` (clamped to CPU count inside
3536+
// the pool) so each DOM worker has a CPU peer for its blend
3537+
// calls; no benefit from oversubscribing. Only allocated when
3538+
// there are actually transition frames to blend.
3539+
let shaderPool: ShaderTransitionWorkerPool | null = null;
34673540
try {
34683541
// Worker 0 reuses the main `domSession`; spawn the rest.
34693542
for (let w = 0; w < workerCanvasesNeeded; w++) {
@@ -3482,6 +3555,25 @@ export async function executeRenderJob(
34823555
const sessions: CaptureSession[] = [domSession, ...workerSessions];
34833556
const activeWorkerCount = sessions.length;
34843557

3558+
// Spawn the shader-blend pool now that we know the actual
3559+
// DOM-worker count. Skipping when there are no transitions
3560+
// avoids ~10–50ms × N worker spawn cost on the SDR
3561+
// non-transition fast path.
3562+
if (hasTransitions) {
3563+
try {
3564+
shaderPool = await createShaderTransitionWorkerPool({
3565+
size: activeWorkerCount,
3566+
log,
3567+
});
3568+
} catch (err) {
3569+
log.warn(
3570+
"[Render] Failed to spawn shader-blend worker pool; falling back to inline shader blend",
3571+
{ error: err instanceof Error ? err.message : String(err) },
3572+
);
3573+
shaderPool = null;
3574+
}
3575+
}
3576+
34853577
// Per-worker normal-frame canvas, allocated once and reused.
34863578
// Worker 0 reuses `normalCanvas` (already allocated above).
34873579
const workerCanvases: Buffer[] = [normalCanvas];
@@ -3546,6 +3638,7 @@ export async function executeRenderJob(
35463638
myTransitionBuffers,
35473639
assertNotAborted,
35483640
nativeHdrIds,
3641+
shaderPool,
35493642
);
35503643
await writeEncoded(i, myTransitionBuffers.output);
35513644
} else {
@@ -3585,6 +3678,13 @@ export async function executeRenderJob(
35853678
});
35863679
});
35873680
}
3681+
if (shaderPool) {
3682+
await shaderPool.terminate().catch((err) => {
3683+
log.warn("Shader-blend worker pool terminate failed", {
3684+
err: err instanceof Error ? err.message : String(err),
3685+
});
3686+
});
3687+
}
35883688
}
35893689
} else {
35903690
// ── Legacy sequential path ─────────────────────────────────────
Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
/**
2+
* Worker entry point for off-main-thread shader-blend execution.
3+
*
4+
* The hf#677 follow-up moved the layered transition pipeline (dual-scene
5+
* seek/mask/screenshot) onto per-worker DOM sessions, but the per-pixel JS
6+
* shader-blend at the tail of `processLayeredTransitionFrame` still ran on
7+
* the orchestrator's main event loop. Complex shaders (`domain-warp`,
8+
* `swirl-vortex`, `glitch`) iterate every pixel of the rgb48le buffer with
9+
* multiple noise/sample calls per pixel — hundreds of milliseconds per call
10+
* — so N concurrent DOM workers all firing shader-blends saturated the
11+
* single Node thread. The empirical worker-count sweep on the #677 fixture
12+
* (w=1=218s, w=2=183s, w=6=184s, w=12=188s) flattens after w=2, which is the
13+
* single-threaded-downstream signature.
14+
*
15+
* This worker runs `TRANSITIONS[shader](from, to, output, w, h, p)` on a
16+
* dedicated Node `worker_threads` Worker. The pool dispatches one frame at
17+
* a time per worker. The rgb48le scratch Buffers are moved in and out via
18+
* `transferList` — zero-copy at the ArrayBuffer level — so the only
19+
* per-frame cost is the postMessage round-trip (~sub-millisecond on the
20+
* 2.4 MB 854×480 buffers) plus the shader-blend itself.
21+
*
22+
* Lifecycle:
23+
*
24+
* 1. Pool constructor spawns N of these workers up front.
25+
* 2. Main thread posts `{ shader, bufferA, bufferB, output, width, height,
26+
* progress }` with `transferList: [bufferA, bufferB, output]`. The three
27+
* ArrayBuffers are detached on the sender; the caller must NOT touch
28+
* them until the worker replies.
29+
* 3. Worker wraps each ArrayBuffer as a Node Buffer view (zero-copy),
30+
* invokes `TRANSITIONS[shader] ?? crossfade`, and posts `{ ok: true,
31+
* output }` back with `transferList: [output]`. (The two input ArrayBuffers
32+
* are also returned so the main thread can re-attach them to the worker's
33+
* `LayeredTransitionBuffers` slot for reuse on the next frame.)
34+
* 4. On unknown shader / runtime exception, worker posts `{ ok: false, error,
35+
* bufferA, bufferB, output }` — all three are still transferred back so
36+
* the caller can release them.
37+
*
38+
* The worker holds no per-frame state. It is shared across DOM-session
39+
* workers and across the entire render — only spawned once at render start
40+
* and terminated at render end.
41+
*/
42+
43+
import { parentPort } from "node:worker_threads";
44+
// Import the shader-blend table from a dedicated `./shader-transitions`
45+
// subpath export of `@hyperframes/engine` rather than the package root.
46+
// Rationale:
47+
//
48+
// 1. `shaderTransitions.ts` is fully self-contained (no internal imports).
49+
// Going through engine's root index pulls in the rest of the engine
50+
// graph, which fails under `worker_threads` + tsx in dev/test: the
51+
// tsx loader's `.js → .ts` rewrite does NOT survive the Worker
52+
// boundary, so internal specifiers like `./config.js` from `index.ts`
53+
// fail to resolve. The subpath sidesteps that by pointing the
54+
// resolver straight at the import-free file.
55+
//
56+
// 2. In the production esbuild bundle (build.mjs entry
57+
// `src/services/shaderTransitionWorker.ts`) the workspace alias plugin
58+
// redirects `@hyperframes/engine/shader-transitions` to the same TS
59+
// source and bundles it inline, so behavior is identical.
60+
import { TRANSITIONS, crossfade } from "@hyperframes/engine/shader-transitions";
61+
62+
interface ShaderJobRequest {
63+
shader: string;
64+
bufferA: ArrayBuffer;
65+
bufferB: ArrayBuffer;
66+
output: ArrayBuffer;
67+
width: number;
68+
height: number;
69+
progress: number;
70+
}
71+
72+
interface ShaderJobOk {
73+
ok: true;
74+
bufferA: ArrayBuffer;
75+
bufferB: ArrayBuffer;
76+
output: ArrayBuffer;
77+
}
78+
79+
interface ShaderJobErr {
80+
ok: false;
81+
error: string;
82+
bufferA: ArrayBuffer;
83+
bufferB: ArrayBuffer;
84+
output: ArrayBuffer;
85+
}
86+
87+
export type ShaderJobResult = ShaderJobOk | ShaderJobErr;
88+
89+
if (!parentPort) {
90+
// Defensive — this module is only meaningful inside a worker_thread.
91+
// If imported on the main thread (e.g. by an accidental top-level test),
92+
// do nothing rather than throwing, so static analysis stays clean.
93+
// eslint-disable-next-line no-console
94+
console.warn("[shaderTransitionWorker] no parentPort; module loaded on main thread");
95+
} else {
96+
parentPort.on("message", (msg: ShaderJobRequest) => {
97+
const { shader, bufferA, bufferB, output, width, height, progress } = msg;
98+
// Re-wrap the transferred ArrayBuffers as Node Buffers. Buffer.from(ab)
99+
// is a zero-copy view over the same underlying memory — no allocation,
100+
// no data copy. The shader functions are typed to take Buffer and use
101+
// its readUInt16LE/writeUInt16LE API.
102+
const bufA = Buffer.from(bufferA);
103+
const bufB = Buffer.from(bufferB);
104+
const out = Buffer.from(output);
105+
106+
try {
107+
const fn = TRANSITIONS[shader] ?? crossfade;
108+
fn(bufA, bufB, out, width, height, progress);
109+
const reply: ShaderJobOk = {
110+
ok: true,
111+
bufferA,
112+
bufferB,
113+
output,
114+
};
115+
parentPort!.postMessage(reply, [bufferA, bufferB, output]);
116+
} catch (err) {
117+
const reply: ShaderJobErr = {
118+
ok: false,
119+
error: err instanceof Error ? err.message : String(err),
120+
bufferA,
121+
bufferB,
122+
output,
123+
};
124+
parentPort!.postMessage(reply, [bufferA, bufferB, output]);
125+
}
126+
});
127+
}

0 commit comments

Comments
 (0)