Skip to content

Commit 1664fe6

Browse files
vanceingallsclaude
andcommitted
fix(core,producer,skills): unicode paths, non-Error rejections, shell callers
Three R3 findings. The redactor's segment classes were ASCII `\w`, so `/数据/客户/秘密视频.mp4` and `/data/客户/secret.mp4` went out verbatim — and the generic redactor also feeds CLI telemetry and producer observation messages, where no known-path list compensates. Segments are now defined by their delimiters instead of an alphabet, which is correct for every script by construction rather than requiring Unicode classes to be kept correct. The bare-relative lookbehind had the same ASCII assumption and let a match start mid-token, redacting `客户/秘密/视频.mp4` to `客户[path]`; it is now a token boundary, and bare-relative runs before absolute so it claims the whole token. sanitizeProbeFailure cast the rejection reason to Error and read `.message`. An injected probe can reject with anything, so `Promise.reject("failed")` gave `undefined` and threw inside the redactor — converting a returned failure result into a rejected promise. Normalized at the boundary, and redactKnownPaths no longer throws on a non-string. The contract only admitted .ts/.js/.mjs/.cjs, so it missed shipped shell and Python callers. frame_strip.sh passed a user-controlled path as ffprobe's last positional with no terminator; render-and-composite.sh had four more. Both fixed, and the sweep now covers .py/.sh. Python list argvs are bracket literals so they get the same position check; shell command lines get a separate presence check, because checking position there needs a shell parser — stated as the weaker guarantee it is rather than implied to be equal. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 6c5403f commit 1664fe6

8 files changed

Lines changed: 169 additions & 19 deletions

File tree

‎packages/core/src/telemetryRedaction.test.ts‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,39 @@ describe("redactTelemetryString", () => {
8686
});
8787
});
8888

89+
// The segment classes were ASCII `\w`, so a non-Latin path went out verbatim.
90+
// This redactor also feeds CLI telemetry and producer observation messages,
91+
// where no known-path list is supplied to compensate.
92+
describe("non-Latin paths", () => {
93+
it.each([
94+
"/数据/客户/秘密视频.mp4",
95+
"/данные/клиент/видео.mp4",
96+
"/data/客户/secret.mp4",
97+
"/Users/alice/проект/видео.mp4",
98+
])("redacts the absolute path %s", (path) => {
99+
const out = redactTelemetryString(`ffprobe failed reading ${path}`);
100+
expect(out).toBe("ffprobe failed reading [path]");
101+
});
102+
103+
it("redacts a non-Latin bare relative path without stranding the first segment", () => {
104+
// The lookbehind used to be `\w`-based, so a match could start mid-token
105+
// when the preceding character was non-ASCII: this redacted to `客户[path]`.
106+
expect(redactTelemetryString("could not open 客户/秘密/视频.mp4")).toBe(
107+
"could not open [path]",
108+
);
109+
});
110+
111+
it.each(["./资产/背景.mp3", "../输出/final.wav"])("redacts the relative path %s", (path) => {
112+
expect(redactTelemetryString(`could not open ${path}`)).toBe("could not open [path]");
113+
});
114+
115+
it("redacts a non-Latin bare basename", () => {
116+
expect(redactTelemetryString("Invalid data found in 秘密视频.mp4")).toBe(
117+
"Invalid data found in [file]",
118+
);
119+
});
120+
});
121+
89122
describe("redactKnownPaths", () => {
90123
// Shape matching has holes by construction. A caller that built the argv
91124
// knows the exact path, so it can name it instead of hoping a regex does.
@@ -113,4 +146,11 @@ describe("redactKnownPaths", () => {
113146
it("ignores paths too short to be distinctive", () => {
114147
expect(redactKnownPaths("a stream at a rate", ["a"])).toBe("a stream at a rate");
115148
});
149+
150+
// This sits on an error path: throwing here turns a reported failure into an
151+
// unhandled rejection, which is what a non-Error rejection's `undefined`
152+
// message caused.
153+
it.each([undefined, null, 42, {}])("returns a string for the non-string input %s", (value) => {
154+
expect(() => redactKnownPaths(value as unknown as string, ["/tmp/x.mp4"])).not.toThrow();
155+
});
116156
});

‎packages/core/src/telemetryRedaction.ts‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,24 @@ function redactUrlQueryStrings(value: string): string {
1515
* prose after it, and a path with a space still gets its remaining segments
1616
* redacted, which is the part that carries the identifying information.
1717
*/
18-
const SEGMENT = String.raw`[\w.\-@+()~]+`;
18+
/**
19+
* A path segment is defined by what ENDS it, not by an alphabet.
20+
*
21+
* `[\w...]` is ASCII-only, so `/数据/客户/秘密视频.mp4` and `/data/客户/secret.mp4`
22+
* passed through completely unredacted — the generic redactor also feeds CLI
23+
* telemetry and producer observation messages, where no known-path list is
24+
* supplied to cover for it. Enumerating Unicode classes instead (`\p{L}\p{N}…`)
25+
* would work but has to be kept correct for marks, joiners and emoji; a
26+
* delimiter-based rule is right for every script by construction.
27+
*
28+
* Whitespace and quotes end a segment; so do the separators themselves.
29+
* Space stays excluded for the original reason: including it would let a match
30+
* run past the path and swallow the prose after it.
31+
*/
32+
const SEGMENT = String.raw`[^\s/\\'"]+`;
1933

2034
/** Same, minus the dot, so a trailing `.ext` can be matched separately. */
21-
const SEGMENT_NODOT = String.raw`[\w\-@+()~]+`;
35+
const SEGMENT_NODOT = String.raw`[^\s/\\'".]+`;
2236

2337
/**
2438
* Once a match is established as a path, consume the rest of the token.
@@ -63,7 +77,7 @@ const RELATIVE_PATH = new RegExp(
6377
* whose host we deliberately keep.
6478
*/
6579
const ASSET_BASENAME =
66-
/(?<![\w/\\])[\w.\-@+()~]+\.(?:mp4|mov|mkv|webm|avi|m4v|mpe?g|ts|mp3|wav|aac|m4a|flac|ogg|opus|png|jpe?g|gif|webp|svg|html?|json|srt|vtt|ass)\b/gi;
80+
/(?<![\w/\\])[^\s/\\'"]+\.(?:mp4|mov|mkv|webm|avi|m4v|mpe?g|ts|mp3|wav|aac|m4a|flac|ogg|opus|png|jpe?g|gif|webp|svg|html?|json|srt|vtt|ass)\b/gi;
6781

6882
/**
6983
* A relative path with NO `./` prefix — `assets/bgm.mp3`,
@@ -80,11 +94,11 @@ const BARE_RELATIVE_PATH = new RegExp(
8094
[
8195
// Two or more separators: `customer/acme/video.mp4`. No extension needed —
8296
// that much structure is already a path.
83-
String.raw`(?<![\w/\\.:@-])(?:${SEGMENT}[\\/]){2,}${SEGMENT}${TOKEN_TAIL}`,
97+
String.raw`(?<![^\s'\"(=,\[])(?:${SEGMENT}[\\/]){2,}${SEGMENT}${TOKEN_TAIL}`,
8498
// One separator, but the last segment carries a file extension:
8599
// `assets/bgm.mp3`. That segment is dot-free on purpose — SEGMENT includes
86100
// `.`, so a greedy one swallows the extension this rule needs.
87-
String.raw`(?<![\w/\\.:@-])${SEGMENT}[\\/]${SEGMENT_NODOT}\.\w{1,8}\b${TOKEN_TAIL}`,
101+
String.raw`(?<![^\s'\"(=,\[])${SEGMENT}[\\/]${SEGMENT_NODOT}\.\w{1,8}\b${TOKEN_TAIL}`,
88102
].join("|"),
89103
"g",
90104
);
@@ -99,6 +113,10 @@ const BARE_RELATIVE_PATH = new RegExp(
99113
* reports only that.
100114
*/
101115
export function redactKnownPaths(value: string, paths: readonly string[]): string {
116+
// Fail soft on a non-string. This sits on an error path, so throwing here
117+
// converts a reported failure into an unhandled rejection — which is exactly
118+
// what happened when a caller passed a non-Error rejection's `.message`.
119+
if (typeof value !== "string") return "";
102120
let out = value;
103121
for (const path of paths) {
104122
if (typeof path !== "string" || path.length === 0) continue;
@@ -120,8 +138,10 @@ function redactFilePaths(value: string): string {
120138
// (`/assets/x.mp3`), so the absolute rule would consume it and leave the
121139
// leading `.` stranded outside the redaction.
122140
.replace(RELATIVE_PATH, "[path]")
123-
.replace(ABSOLUTE_PATH, "[path]")
141+
// Bare-relative BEFORE absolute: a bare path's interior satisfies the
142+
// absolute rule, which would claim it and strand the first segment.
124143
.replace(BARE_RELATIVE_PATH, "[path]")
144+
.replace(ABSOLUTE_PATH, "[path]")
125145
.replace(ASSET_BASENAME, "[file]")
126146
);
127147
}

‎packages/producer/src/services/render/audioPadTrim.test.ts‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,4 +337,45 @@ describe("PadTrimAudioResult.error never carries the input path", () => {
337337
expect(result.error ?? "").not.toContain("acme-secret");
338338
expect(result.error ?? "").toContain("failed to probe audio");
339339
});
340+
341+
// An injected probe can reject with anything. Casting the reason to Error and
342+
// reading `.message` yielded undefined, which threw inside the redactor and
343+
// turned a returned failure result into a rejected promise.
344+
describe("a probe that rejects with a non-Error value", () => {
345+
const nonErrors: Array<[string, unknown]> = [
346+
["a string", "probe failed"],
347+
["undefined", undefined],
348+
["null", null],
349+
["a number", 42],
350+
["a plain object", { code: "ENOENT" }],
351+
];
352+
353+
for (const [label, reason] of nonErrors) {
354+
it(`still returns a failed result when the video probe rejects with ${label}`, async () => {
355+
const result = await padOrTrimAudioToVideoFrameCount({
356+
videoPath: "/data/acme-secret/video.mp4",
357+
audioPath: "/tmp/audio.aac",
358+
outputPath: "/tmp/out.aac",
359+
probeVideoFrameInfo: () => Promise.reject(reason),
360+
probeAudioInfo: () => Promise.resolve({ durationSeconds: 1 }),
361+
runFfmpeg: () => Promise.resolve({ success: true }),
362+
});
363+
expect(result.success).toBe(false);
364+
expect(result.error ?? "").toContain("failed to probe video");
365+
});
366+
367+
it(`still returns a failed result when the audio probe rejects with ${label}`, async () => {
368+
const result = await padOrTrimAudioToVideoFrameCount({
369+
videoPath: "/tmp/v.mp4",
370+
audioPath: "/data/acme-secret/audio.aac",
371+
outputPath: "/tmp/out.aac",
372+
probeVideoFrameInfo: () => Promise.resolve({ frameCount: 30, fpsNum: 30, fpsDen: 1 }),
373+
probeAudioInfo: () => Promise.reject(reason),
374+
runFfmpeg: () => Promise.resolve({ success: true }),
375+
});
376+
expect(result.success).toBe(false);
377+
expect(result.error ?? "").toContain("failed to probe audio");
378+
});
379+
}
380+
});
340381
});

‎packages/producer/src/services/render/audioPadTrim.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -230,7 +230,13 @@ function formatSeconds(sec: number): string {
230230
* recognise them), then the generic shape-based scrub for anything the message
231231
* picked up elsewhere.
232232
*/
233-
function sanitizeProbeFailure(message: string, paths: readonly string[]): string {
233+
function sanitizeProbeFailure(reason: unknown, paths: readonly string[]): string {
234+
// Normalized here, not at the call sites. A caller-supplied probe can reject
235+
// with anything — `Promise.reject("probe failed")` has no `.message`, so
236+
// casting to Error yielded `undefined` and threw inside the redactor. That
237+
// turned a returned failure result into a rejected promise, which is a
238+
// behaviour regression the cast introduced.
239+
const message = reason instanceof Error ? reason.message : String(reason);
234240
return redactTelemetryString(redactKnownPaths(message, paths));
235241
}
236242

@@ -261,7 +267,7 @@ export async function padOrTrimAudioToVideoFrameCount(
261267
0,
262268
audioResult.status === "fulfilled" ? audioResult.value.durationSeconds : 0,
263269
`audioPadTrim: failed to probe video: ${sanitizeProbeFailure(
264-
(videoResult.reason as Error).message,
270+
videoResult.reason,
265271
probePaths,
266272
)}`,
267273
);
@@ -272,7 +278,7 @@ export async function padOrTrimAudioToVideoFrameCount(
272278
0,
273279
0,
274280
`audioPadTrim: failed to probe audio: ${sanitizeProbeFailure(
275-
(audioResult.reason as Error).message,
281+
audioResult.reason,
276282
probePaths,
277283
)}`,
278284
);

‎packages/producer/src/utils/ffprobeArgvContract.test.ts‎

Lines changed: 46 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,20 @@ const REPO_ROOT = join(import.meta.dirname, "..", "..", "..", "..");
3232
const SWEEP_ROOTS = ["packages", "skills", "scripts"];
3333

3434
/** `.mjs`/`.cjs` are first-class here — the skill scripts are not TypeScript. */
35-
const SOURCE_EXT = /\.(?:ts|mjs|cjs|js)$/;
35+
const SOURCE_EXT = /\.(?:ts|mjs|cjs|js|py|sh)$/;
36+
37+
/**
38+
* Shell-syntax invocations, checked separately and more weakly.
39+
*
40+
* A JS/Python argv is a bracketed literal, so the parser above can check that
41+
* `--` is the PENULTIMATE entry. A shell command line is not a literal —
42+
* `ffprobe -v error ... "$BG" 2>/dev/null | tr -dc '0-9.'` has redirections,
43+
* pipes and substitutions after the input — so checking position would need a
44+
* shell parser. This asserts the terminator is PRESENT on any ffprobe command
45+
* line, which is weaker but is the part that was missing, and it is honest
46+
* about being weaker rather than implying the same guarantee.
47+
*/
48+
const SHELL_EXT = /\.sh$/;
3649

3750
/**
3851
* The caller set as of the sweep that introduced this contract.
@@ -139,11 +152,15 @@ function isSourceFile(entry: string): boolean {
139152
return true;
140153
}
141154

142-
function discoverCallers(): { found: string[]; unclassified: string[] } {
155+
function discoverCallers(): { found: string[]; unclassified: string[]; shell: string[] } {
143156
const found: string[] = [];
144157
const unclassified: string[] = [];
158+
const shell: string[] = [];
145159
const classify = (abs: string): void => {
146160
const src = readFileSync(abs, "utf8");
161+
if (SHELL_EXT.test(abs) && /(?:^|[^\w-])ffprobe\s+-/m.test(src)) {
162+
shell.push(relative(REPO_ROOT, abs));
163+
}
147164
// Discovery is ARGV-shaped, not call-shaped. Matching on spawn/execFile
148165
// misses a dependency-injected runner — `runner("ffprobe", [...])` in
149166
// studio-server's mediaValidation.ts is exactly that, and a call-shaped
@@ -168,7 +185,7 @@ function discoverCallers(): { found: string[]; unclassified: string[] } {
168185
/* root absent in a partial checkout */
169186
}
170187
}
171-
return { found: found.sort(), unclassified: unclassified.sort() };
188+
return { found: found.sort(), unclassified: unclassified.sort(), shell: shell.sort() };
172189
}
173190

174191
/**
@@ -211,6 +228,32 @@ function argvTails(source: string): Array<{ snippet: string; tail: string[] }> {
211228
return tails;
212229
}
213230

231+
describe("shell ffprobe invocations terminate their options", () => {
232+
const shellFiles = discoverCallers().shell;
233+
234+
it("finds the shell callers", () => {
235+
// Guards the guard: `frame_strip.sh` and `render-and-composite.sh` both
236+
// shipped un-terminated while the JS-only sweep reported the class closed.
237+
expect(shellFiles.length).toBeGreaterThan(0);
238+
});
239+
240+
it.each(shellFiles)("%s passes -- on every ffprobe command line", (relPath) => {
241+
const source = readFileSync(join(REPO_ROOT, relPath), "utf8");
242+
const offenders = source
243+
.split("\n")
244+
.map((line, index) => ({ line: line.trim(), number: index + 1 }))
245+
// An invocation passes flags. `command -v ffprobe >/dev/null` is a PATH
246+
// check and `echo "ffmpeg/ffprobe not on PATH"` is a message; neither
247+
// takes an input, and both were reported before this narrowed.
248+
.filter(({ line }) => /(?:^|[^\w-])ffprobe\s+-/.test(line) && !line.startsWith("#"))
249+
.filter(({ line }) => !/\b(?:command\s+-v|which|type)\s+ffprobe/.test(line))
250+
.filter(({ line }) => !/\s--\s/.test(line))
251+
.map(({ line, number }) => `${number}: ${line.slice(0, 80)}`);
252+
253+
expect(offenders, `${relPath}: ffprobe command lines missing "--"`).toEqual([]);
254+
});
255+
});
256+
214257
describe("ffprobe argv contract", () => {
215258
const { found: callers, unclassified } = discoverCallers();
216259

‎skills-manifest.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"source": "heygen-com/hyperframes",
33
"skills": {
44
"embedded-captions": {
5-
"hash": "8e8bd824567c3e17",
5+
"hash": "e8c2c3b6dfd04b39",
66
"files": 140
77
},
88
"faceless-explainer": {
@@ -66,7 +66,7 @@
6666
"files": 28
6767
},
6868
"remotion-to-hyperframes": {
69-
"hash": "3a0e6c2affb9f74e",
69+
"hash": "3ecc684432b298dd",
7070
"files": 70
7171
},
7272
"slideshow": {

‎skills/embedded-captions/scripts/render-and-composite.sh‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -352,8 +352,8 @@ else
352352
fi
353353

354354
# Probe render dims for ffmpeg scale
355-
W="$(ffprobe -v error -select_streams v:0 -show_entries stream=width -of default=nw=1:nk=1 "$BG")"
356-
H="$(ffprobe -v error -select_streams v:0 -show_entries stream=height -of default=nw=1:nk=1 "$BG")"
355+
W="$(ffprobe -v error -select_streams v:0 -show_entries stream=width -of default=nw=1:nk=1 -- "$BG")"
356+
H="$(ffprobe -v error -select_streams v:0 -show_entries stream=height -of default=nw=1:nk=1 -- "$BG")"
357357

358358
# Clamp every composite to the matte (= source-video) length. The render uses
359359
# plan.duration / data-duration, which can exceed the source (e.g. Whisper word
@@ -365,7 +365,7 @@ H="$(ffprobe -v error -select_streams v:0 -show_entries stream=height -of defaul
365365
# the a-roll stream ends). The true a-roll duration is authoritative: clamp to
366366
# min(matte frames / fps, source duration).
367367
MATTE_DUR="$(awk "BEGIN{printf \"%.3f\", $(ls "$PROJECT/frames_fg" | wc -l)/$FPS}")"
368-
SRC_DUR="$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$PROJECT/source.mp4" 2>/dev/null || true)"
368+
SRC_DUR="$(ffprobe -v error -show_entries format=duration -of csv=p=0 -- "$PROJECT/source.mp4" 2>/dev/null || true)"
369369
if [[ -n "${SRC_DUR:-}" ]]; then
370370
MATTE_DUR="$(awk "BEGIN{m=$MATTE_DUR; s=$SRC_DUR; printf \"%.3f\", (s>0 && s<m) ? s : m}")"
371371
fi
@@ -378,7 +378,7 @@ echo "[render] clamp output to source/matte length: ${MATTE_DUR}s"
378378
# last-caption time instead of the clip length. Clamp to the bg length so we never
379379
# ship the only-foreground tail, and tell the author the real fix.
380380
if [[ -f "$BG" ]]; then
381-
BG_DUR="$(ffprobe -v error -show_entries format=duration -of default=nokey=1:noprint_wrappers=1 "$BG" 2>/dev/null | tr -dc '0-9.')"
381+
BG_DUR="$(ffprobe -v error -show_entries format=duration -of default=nokey=1:noprint_wrappers=1 -- "$BG" 2>/dev/null | tr -dc '0-9.')"
382382
if [[ -n "$BG_DUR" ]] && awk "BEGIN{exit !($BG_DUR < $MATTE_DUR - 0.3)}"; then
383383
echo "[render] ⚠ background plate is ${BG_DUR}s but the clip is ${MATTE_DUR}s — the composition is shorter than the footage." >&2
384384
echo " The tail would show ONLY the foreground subject on black. FIX: set the composition" >&2

‎skills/remotion-to-hyperframes/scripts/frame_strip.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ probe = subprocess.run(
5353
["ffprobe", "-v", "error", "-select_streams", "v:0",
5454
"-show_entries", "stream=r_frame_rate,nb_read_frames,duration",
5555
"-show_entries", "format=duration",
56-
"-of", "json", "-count_frames", baseline],
56+
"-of", "json", "-count_frames", "--", baseline],
5757
check=True, capture_output=True, text=True,
5858
)
5959
data = json.loads(probe.stdout)

0 commit comments

Comments
 (0)