Skip to content

Commit 4dbf0d9

Browse files
vanceingallsclaude
andcommitted
fix(producer): fail closed when no live element count is available
R4 review finding, and the comment I wrote in R3 was simply wrong: the probe session is NOT running for every render. probeStage's needsBrowser gate launches one only for unknown duration, unresolved compositions, or specific media cases — and hasRuntimeInsertedMedia matches only createElement("video"|"audio"), never createElement("span"). So the exact shape that motivated the live-DOM fix (a known-duration, media-free caption comp building thousands of nodes in script) gets NO probe, falls back to the static source scan, reads as ~2 elements, and could enter the applied cohort at 40k live nodes. The R3 fix measured the right thing but only for the population that already had a probe. Now the count carries provenance and the band fails closed: - resolveCompositionElementCount returns { count, source: "live" | "static" }. Only "live" — an actual DOM measurement — may open the band. - resolveDeShortBand gains a third decisive outcome, "unmeasured", for the static case. It deliberately does NOT report skipped_elements: a static undercount is not a real oversize observation, and putting it in the control arm would contaminate the DiD just as putting it in the treatment arm would. Neither cohort; never routes. - composition_element_count_source ships alongside the count, so the fleet rate of "static" sizes the population a future conditional-probe-launch would unlock — which is the data PR B needs to decide whether that launch cost is worth paying. Regression coverage walks the real chain rather than a full render, using the production functions in pipeline order: probeRequiresBrowser (newly extracted from the inline needsBrowser expression, so the gate is testable at all) returns false for the caption-comp shape → the resolver reports static and a count under the ceiling → the band reports unmeasured, not applied. Fault injection confirms it bites: removing the one guard line fails exactly these three tests. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent def98f7 commit 4dbf0d9

8 files changed

Lines changed: 259 additions & 50 deletions

File tree

‎packages/cli/src/commands/render.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1433,6 +1433,7 @@ function trackRenderMetrics(
14331433
deWorkerInversion: perf?.drawElement?.workerInversion,
14341434
dePreInversionWorkers: perf?.drawElement?.preInversionWorkers,
14351435
compositionElementCount: perf?.drawElement?.compositionElementCount,
1436+
compositionElementCountSource: perf?.drawElement?.compositionElementCountSource,
14361437
deShortBand: perf?.drawElement?.shortBand,
14371438
deParallelRouter: perf?.drawElement?.parallelRouter,
14381439
dePreRouterWorkers: perf?.drawElement?.preRouterWorkers,

‎packages/cli/src/telemetry/events.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ export interface RenderObservabilityTelemetryPayload {
7373
captureDeWorkerInversion?: string;
7474
captureDePreInversionWorkers?: number;
7575
captureCompositionElementCount?: number;
76+
captureCompositionElementCountSource?: string;
7677
captureDeShortBand?: string;
7778
captureDeParallelRouter?: string;
7879
captureDeGpuRenderer?: string;
@@ -133,6 +134,7 @@ function renderObservabilityEventProperties(props: RenderObservabilityTelemetryP
133134
de_worker_inversion: props.captureDeWorkerInversion,
134135
de_pre_inversion_workers: props.captureDePreInversionWorkers,
135136
composition_element_count: props.captureCompositionElementCount,
137+
composition_element_count_source: props.captureCompositionElementCountSource,
136138
de_short_band: props.captureDeShortBand,
137139
de_parallel_router: props.captureDeParallelRouter,
138140
gpu_renderer: props.captureDeGpuRenderer,
@@ -209,6 +211,7 @@ export function trackRenderComplete(
209211
deWorkerInversion?: string;
210212
dePreInversionWorkers?: number;
211213
compositionElementCount?: number;
214+
compositionElementCountSource?: string;
212215
deShortBand?: string;
213216
deParallelRouter?: string;
214217
dePreRouterWorkers?: number;
@@ -310,6 +313,7 @@ export function trackRenderComplete(
310313
de_worker_inversion: props.deWorkerInversion,
311314
de_pre_inversion_workers: props.dePreInversionWorkers,
312315
composition_element_count: props.compositionElementCount,
316+
composition_element_count_source: props.compositionElementCountSource,
313317
de_short_band: props.deShortBand,
314318
de_parallel_router: props.deParallelRouter,
315319
de_pre_router_workers: props.dePreRouterWorkers,

‎packages/cli/src/telemetry/renderObservability.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ export function renderObservabilityTelemetryPayload(
4343
captureDeWorkerInversion: capture.deWorkerInversion,
4444
captureDePreInversionWorkers: capture.dePreInversionWorkers,
4545
captureCompositionElementCount: capture.compositionElementCount,
46+
captureCompositionElementCountSource: capture.compositionElementCountSource,
4647
captureDeShortBand: capture.deShortBand,
4748
captureDeParallelRouter: capture.deParallelRouter,
4849
captureDeGpuRenderer: capture.deGpuRenderer,

‎packages/producer/src/services/render/observability.ts‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,15 @@ export interface RenderCaptureObservability {
8484
* threshold from real content instead of synthetic sweeps.
8585
*/
8686
compositionElementCount?: number;
87+
/**
88+
* Provenance of `compositionElementCount`: "live" (measured from the probe
89+
* session's real DOM — sees runtime-generated elements) or "static" (source
90+
* markup scan, which does not). The probe is CONDITIONAL, so this is not a
91+
* detail: only a `live` count may open the short band, and the fleet rate of
92+
* "static" sizes the population a future conditional-probe-launch would
93+
* unlock for the band.
94+
*/
95+
compositionElementCountSource?: "live" | "static";
8796
/**
8897
* Short-comp band decision, emitted only when the band is DECISIVE — every
8998
* other inversion-eligibility condition passed and only the floor (250 vs
@@ -97,7 +106,7 @@ export interface RenderCaptureObservability {
97106
* renders form the concurrent control for the difference-in-differences
98107
* read — that is the entire point of the field.
99108
*/
100-
deShortBand?: "applied" | "skipped_elements";
109+
deShortBand?: "applied" | "skipped_elements" | "unmeasured";
101110
/** DE parallel-router outcome: "routed" (fired, held) | "reverted" (fired, self-verify retry rolled back). */
102111
deParallelRouter?: "routed" | "reverted";
103112
/**

‎packages/producer/src/services/render/perfSummary.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,10 @@ export interface DrawElementPerfInput {
8282
preInversionWorkers?: number;
8383
/** Rough compiled-composition element count — gate variable for the short-comp inversion band. */
8484
compositionElementCount?: number;
85+
/** Provenance of the element count: "live" (probe DOM, trusted to gate) | "static" (source scan, not). */
86+
compositionElementCountSource?: "live" | "static";
8587
/** Short-comp band decision when the band was DECISIVE: "applied" (inverts once HF_DE_SHORT_BAND_ROUTE is on; counterfactual in the baseline release) | "skipped_elements" (element ceiling was the only blocker); unset when the band could not have affected this render. */
86-
shortBand?: "applied" | "skipped_elements";
88+
shortBand?: "applied" | "skipped_elements" | "unmeasured";
8789
parallelRouter?: "routed" | "reverted";
8890
/** Auto-resolved worker count before the router pinned it to 3 (set only when the router fired). */
8991
preRouterWorkers?: number;
@@ -126,6 +128,7 @@ function aggregateDrawElement(
126128
workerInversion: de.workerInversion ?? "none",
127129
preInversionWorkers: de.preInversionWorkers,
128130
compositionElementCount: de.compositionElementCount,
131+
compositionElementCountSource: de.compositionElementCountSource,
129132
shortBand: de.shortBand,
130133
parallelRouter: de.parallelRouter ?? "none",
131134
preRouterWorkers: de.preRouterWorkers,

‎packages/producer/src/services/render/stages/probeStage.ts‎

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,37 @@ function hasRuntimeInsertedMedia(html: string): boolean {
184184
);
185185
}
186186

187+
/**
188+
* Does this render need a browser probe at all?
189+
*
190+
* Extracted as a pure predicate because whether a probe runs decides whether
191+
* a LIVE DOM element count is available downstream, and the short-comp
192+
* inversion band fails closed without one (see
193+
* `resolveCompositionElementCount` / `resolveDeShortBand`). Notably NONE of
194+
* these conditions fire for a known-duration, media-free composition that
195+
* builds thousands of `div`/`span` nodes in its own init script —
196+
* `hasRuntimeInsertedMedia` matches only `createElement("video"|"audio")` —
197+
* so that shape is measured statically and must never reach the band's
198+
* `applied` cohort (review finding, R4).
199+
*/
200+
export function probeRequiresBrowser(args: {
201+
durationSeconds: number;
202+
unresolvedCompositionCount: number;
203+
hasAutoStart: boolean;
204+
hasScriptedAudio: boolean;
205+
hasVariableMedia: boolean;
206+
hasInsertedMedia: boolean;
207+
}): boolean {
208+
return (
209+
args.durationSeconds <= 0 ||
210+
args.unresolvedCompositionCount > 0 ||
211+
args.hasAutoStart ||
212+
args.hasScriptedAudio ||
213+
args.hasVariableMedia ||
214+
args.hasInsertedMedia
215+
);
216+
}
217+
187218
export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageResult> {
188219
const {
189220
projectDir,
@@ -217,13 +248,14 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
217248
);
218249
const hasVariableMedia = hasVariableBoundMedia(compiled.html, job.config.variables);
219250
const hasInsertedMedia = hasRuntimeInsertedMedia(compiled.html);
220-
const needsBrowser =
221-
composition.duration <= 0 ||
222-
compiled.unresolvedCompositions.length > 0 ||
223-
hasAutoStart ||
224-
hasScriptedAudio ||
225-
hasVariableMedia ||
226-
hasInsertedMedia;
251+
const needsBrowser = probeRequiresBrowser({
252+
durationSeconds: composition.duration,
253+
unresolvedCompositionCount: compiled.unresolvedCompositions.length,
254+
hasAutoStart,
255+
hasScriptedAudio,
256+
hasVariableMedia,
257+
hasInsertedMedia,
258+
});
227259

228260
if (needsBrowser) {
229261
const reasons = [];

‎packages/producer/src/services/renderOrchestrator.test.ts‎

Lines changed: 139 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ import {
4444
shouldStreamParallelCapture,
4545
shouldUseStreamingEncode,
4646
} from "./renderOrchestrator.js";
47+
import { probeRequiresBrowser } from "./render/stages/probeStage.js";
4748
import { ensureFrameWritten } from "./render/stages/captureHdrFrameShared.js";
4849
import { resolveCompositeTransfer, shouldUseLayeredComposite } from "./hdrCompositor.js";
4950
import {
@@ -1765,9 +1766,12 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
17651766
});
17661767

17671768
describe("resolveDeShortBand", () => {
1768-
it("reports applied only when decisive and the element ceiling cleared", () => {
1769+
const live = { elementCountSource: "live" as const };
1770+
1771+
it("reports applied only when decisive, live-measured, and under the ceiling", () => {
17691772
expect(
17701773
resolveDeShortBand({
1774+
...live,
17711775
invertAtBaseFloor: false,
17721776
invertAtBandFloor: true,
17731777
bandEnabled: true,
@@ -1776,9 +1780,10 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
17761780
).toBe("applied");
17771781
});
17781782

1779-
it("reports skipped_elements when decisive but the comp is oversized", () => {
1783+
it("reports skipped_elements when live-measured and over the ceiling", () => {
17801784
expect(
17811785
resolveDeShortBand({
1786+
...live,
17821787
invertAtBaseFloor: false,
17831788
invertAtBandFloor: true,
17841789
bandEnabled: true,
@@ -1790,6 +1795,7 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
17901795
it("is undefined when the base floor already inverts — the band changed nothing", () => {
17911796
expect(
17921797
resolveDeShortBand({
1798+
...live,
17931799
invertAtBaseFloor: true,
17941800
invertAtBandFloor: true,
17951801
bandEnabled: true,
@@ -1801,6 +1807,7 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
18011807
it("is undefined when neither floor inverts — the render was ineligible for some other reason", () => {
18021808
expect(
18031809
resolveDeShortBand({
1810+
...live,
18041811
invertAtBaseFloor: false,
18051812
invertAtBandFloor: false,
18061813
bandEnabled: true,
@@ -1816,13 +1823,113 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
18161823
it("HF_DE_SHORT_MAX_ELEMENTS=0 (bandEnabled=false) reports undefined even when the render would otherwise be decisive", () => {
18171824
expect(
18181825
resolveDeShortBand({
1826+
...live,
18191827
invertAtBaseFloor: false,
18201828
invertAtBandFloor: true, // an otherwise-eligible in-band render
18211829
bandEnabled: false, // the kill switch
18221830
bandOpen: false, // deShortBandOpen also false when the switch is off
18231831
}),
18241832
).toBeUndefined();
18251833
});
1834+
1835+
// Review finding (R4): the probe supplying the live count is conditional,
1836+
// so a static count is an UNBOUNDED undercount on runtime-generated DOM.
1837+
// It must never produce "applied" (would route a 40k-node comp) and must
1838+
// never produce "skipped_elements" either (would contaminate the DiD
1839+
// control cohort with a number that isn't a real oversize observation).
1840+
it("fails closed to unmeasured when the count came from the static scan, never applied", () => {
1841+
expect(
1842+
resolveDeShortBand({
1843+
elementCountSource: "static",
1844+
invertAtBaseFloor: false,
1845+
invertAtBandFloor: true,
1846+
bandEnabled: true,
1847+
bandOpen: true, // static scan said "small" — must NOT be believed
1848+
}),
1849+
).toBe("unmeasured");
1850+
});
1851+
1852+
it("reports unmeasured (not skipped_elements) for a static count over the ceiling — it is not a real observation either", () => {
1853+
expect(
1854+
resolveDeShortBand({
1855+
elementCountSource: "static",
1856+
invertAtBaseFloor: false,
1857+
invertAtBandFloor: true,
1858+
bandEnabled: true,
1859+
bandOpen: false,
1860+
}),
1861+
).toBe("unmeasured");
1862+
});
1863+
1864+
it("stays undefined for a static count when the band was not decisive anyway", () => {
1865+
expect(
1866+
resolveDeShortBand({
1867+
elementCountSource: "static",
1868+
invertAtBaseFloor: true,
1869+
invertAtBandFloor: true,
1870+
bandEnabled: true,
1871+
bandOpen: true,
1872+
}),
1873+
).toBeUndefined();
1874+
});
1875+
});
1876+
1877+
// Review finding (R4), end-to-end: the review asked for a regression with a
1878+
// known duration, no media / unresolved compositions, and >2500
1879+
// script-created nodes, asserting it cannot enter `applied` without a live
1880+
// count. This walks the real decision chain rather than a full render —
1881+
// the probe gate, the count resolver, and the band attribution are each the
1882+
// production function, wired in the same order the pipeline wires them.
1883+
describe("no-probe dynamic-DOM composition cannot enter the applied cohort (R4 regression)", () => {
1884+
// A caption-style comp: known duration, no media, builds 4000 spans in
1885+
// its own init script. Mirrors packages/producer/tests/style-10-prod.
1886+
const DYNAMIC_DOM_HTML = [
1887+
'<div id="root"><div id="captions"></div></div>',
1888+
"<script>",
1889+
' const c = document.getElementById("captions");',
1890+
" for (let i = 0; i < 4000; i++) {",
1891+
' const el = document.createElement("span");',
1892+
" el.textContent = String(i);",
1893+
" c.appendChild(el);",
1894+
" }",
1895+
"</script>",
1896+
].join("\n");
1897+
1898+
it("gets no browser probe — none of the probe conditions fire for this shape", () => {
1899+
expect(
1900+
probeRequiresBrowser({
1901+
durationSeconds: 13.3, // known
1902+
unresolvedCompositionCount: 0, // resolved
1903+
hasAutoStart: false, // no media
1904+
hasScriptedAudio: false,
1905+
hasVariableMedia: false,
1906+
// createElement("span") is not createElement("video"|"audio")
1907+
hasInsertedMedia: false,
1908+
}),
1909+
).toBe(false);
1910+
});
1911+
1912+
it("therefore measures statically, and the static count wildly understates the live DOM", async () => {
1913+
const resolved = await resolveCompositionElementCount(null, DYNAMIC_DOM_HTML);
1914+
expect(resolved.source).toBe("static");
1915+
// Source markup has a handful of tags; the live DOM would have 4000+.
1916+
expect(resolved.count).toBeLessThan(2500);
1917+
});
1918+
1919+
it("and therefore reports unmeasured — never applied — so it cannot route or join either cohort", async () => {
1920+
const { count, source } = await resolveCompositionElementCount(null, DYNAMIC_DOM_HTML);
1921+
const bandOpen = source === "live" && count <= 2500;
1922+
const band = resolveDeShortBand({
1923+
// A 400-frame render that would otherwise be perfectly eligible.
1924+
invertAtBaseFloor: false,
1925+
invertAtBandFloor: true,
1926+
bandEnabled: true,
1927+
bandOpen,
1928+
elementCountSource: source,
1929+
});
1930+
expect(band).toBe("unmeasured");
1931+
expect(band).not.toBe("applied");
1932+
});
18261933
});
18271934

18281935
describe("mergeWorkerInitObservability", () => {
@@ -1902,24 +2009,36 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
19022009
// composition's own script creates at runtime (document.createElement) —
19032010
// an unbounded undercount no regex can close. style-10-prod's real
19042011
// per-transcript-word caption generator is exactly this shape: 2 source
1905-
// tags, thousands of live nodes after init. These pin the fix: prefer the
1906-
// live count from an initialized probe session, static scan only as
1907-
// fallback.
2012+
// tags, thousands of live nodes after init.
2013+
//
2014+
// Review finding (R4): the probe that provides the live count is
2015+
// CONDITIONAL, so the fallback cases below are not merely "less precise"
2016+
// — they are UNSAFE to gate on, and every one of them must report
2017+
// provenance "static" so the caller can fail closed.
19082018
it("uses the live DOM count from an initialized probe session, ignoring the (much smaller) source scan", async () => {
19092019
const session = { isInitialized: true, page: { evaluate: async () => 40001 } };
1910-
expect(await resolveCompositionElementCount(session, "<div><span></span></div>")).toBe(40001);
2020+
expect(await resolveCompositionElementCount(session, "<div><span></span></div>")).toEqual({
2021+
count: 40001,
2022+
source: "live",
2023+
});
19112024
});
19122025

1913-
it("falls back to the static scan when there is no probe session", async () => {
1914-
expect(await resolveCompositionElementCount(null, "<div><span></span></div>")).toBe(2);
2026+
it("reports static provenance when there is no probe session", async () => {
2027+
expect(await resolveCompositionElementCount(null, "<div><span></span></div>")).toEqual({
2028+
count: 2,
2029+
source: "static",
2030+
});
19152031
});
19162032

1917-
it("falls back to the static scan when the probe session is not yet initialized", async () => {
2033+
it("reports static provenance when the probe session is not yet initialized", async () => {
19182034
const session = { isInitialized: false, page: { evaluate: async () => 999 } };
1919-
expect(await resolveCompositionElementCount(session, "<div></div>")).toBe(1);
2035+
expect(await resolveCompositionElementCount(session, "<div></div>")).toEqual({
2036+
count: 1,
2037+
source: "static",
2038+
});
19202039
});
19212040

1922-
it("falls back to the static scan when page.evaluate throws (detached frame, mid-navigation)", async () => {
2041+
it("reports static provenance when page.evaluate throws (detached frame, mid-navigation)", async () => {
19232042
const session = {
19242043
isInitialized: true,
19252044
page: {
@@ -1928,12 +2047,18 @@ describe("shouldPreferSingleWorkerDrawElement (DE priority inversion)", () => {
19282047
},
19292048
},
19302049
};
1931-
expect(await resolveCompositionElementCount(session, "<div><span></span></div>")).toBe(2);
2050+
expect(await resolveCompositionElementCount(session, "<div><span></span></div>")).toEqual({
2051+
count: 2,
2052+
source: "static",
2053+
});
19322054
});
19332055

1934-
it("falls back to the static scan when evaluate resolves a non-finite value", async () => {
2056+
it("reports static provenance when evaluate resolves a non-finite value", async () => {
19352057
const session = { isInitialized: true, page: { evaluate: async () => Number.NaN } };
1936-
expect(await resolveCompositionElementCount(session, "<div></div>")).toBe(1);
2058+
expect(await resolveCompositionElementCount(session, "<div></div>")).toEqual({
2059+
count: 1,
2060+
source: "static",
2061+
});
19372062
});
19382063
});
19392064

0 commit comments

Comments
 (0)