Skip to content

Commit 839ae42

Browse files
jmoran-heygenjakemoran2011claudemiguel-heygen
authored
feat(studio): typed audio values, clip fades, audible video drops (#4250)
* feat(studio): typed audio values, clip fades, audible video drops Audio rows in the media inspector (Volume, Speed, Media start) take a typed value: click the readout, type "-6 dB", "1.5x" or "2.5s", Enter. Speed presets are hidden on <audio> clips. Clip-edge fades: data-fade-in / data-fade-out in seconds, honoured by the preview runtime (per-tick gain), the render mixer (afade after volume) and Studio (Fade in / Fade out rows, and drag dots at the top corners of audio clips and audible videos). Linear on both sides so preview and render agree; fades that outrun the clip scale to meet inside it. Dropped videos with an audio stream now land with data-has-audio="true" instead of muted, so the mixer keeps their sound. The media metadata endpoint reports hasAudio from the ffprobe call it already makes. Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix(audio): one owner for fades, dB, and afade-after-volume Illegal fade values become no-fade at fadeGain; typed inspector input still returns null so the readout stays red. Master and group mixes share one filter chain with volume then afade. Fade-out stays anchored to the trimmed clip end. * fix(studio): share audio helpers and bound typed media values * fix(studio): reserve sibling fade duration when editing --------- Co-authored-by: Jake Moran <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Miguel Angel Simon Sierra <[email protected]>
1 parent 3091dcb commit 839ae42

35 files changed

Lines changed: 1582 additions & 54 deletions

‎docs/reference/html-schema.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,7 @@ Audio has no visual lifecycle.
126126
| `data-media-start` / `data-playback-start` | Video, audio, nested composition | Offset into the source file, used by trim and split. Two groups of readers disagree, so the right name depends on the element. **Read only `data-media-start`:** the timing compiler, the HTML parser, `hyperframes validate` (which only inspects `<audio>`), and the engine's audio mixer (which feeds ffmpeg `-ss`). **Read `data-playback-start` first, falling back to `data-media-start`:** the runtime player, Studio (which also writes it), and `hyperframes snapshot`. Because the audio mixer reads only `data-media-start`, a `<video>` authored with just `data-playback-start` renders a trimmed picture over untrimmed audio. Set the name by kind: **`<video>` / `<audio>` → `data-media-start`**; **nested composition → `data-playback-start`** — composition hosts are inspected only by the playback-start-first readers (the media-start-only ones are all `<video>`/`<audio>`-scoped). `data-media-start` still works there as a fallback, but `data-playback-start` is the canonical name Studio writes and normalises to for new composition hosts, so the other name works until an edit rewrites it (it is the [child-timeline offset](/concepts/compositions)). |
127127
| `data-playback-rate` | Video, audio, nested composition | Playback multiplier from `0.1` to `10`. A `rate` lane in `data-automation` overrides it; see [Speed ramps](/reference/speed-ramps) |
128128
| `data-volume` | Video and audio | Static gain. `1` is 0 dB, `0` is silence, and values above `1` boost up to `3.98` (+12 dB) |
129+
| `data-fade-in` / `data-fade-out` | Video and audio | Clip-edge fades in seconds: gain rises 0 → 1 over the first `data-fade-in` seconds and falls 1 → 0 over the last `data-fade-out` seconds of the clip. Linear, anchored to the clip's own edges (a fade-out follows a later trim), and multiplied on top of `data-volume` and any volume lane. Fades that together exceed the clip are scaled to meet inside it. Studio drags them from the dots at an audio clip's top corners or types them in the inspector's Fade rows |
129130
| `data-has-audio="true"` | Video | Declares that the video contributes audio |
130131

131132
Video and audio may omit `data-duration` when their intrinsic duration is known

‎packages/cli/src/docs/data-attributes.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ Core attributes for controlling element timing and behavior.
1111
## Media
1212

1313
- `data-media-start="2"` — Media playback offset / trim point (seconds)
14-
- `data-volume="0.8"` — Audio/video gain. `1` is 0 dB, `0` is silence, and values above `1` boost up to `3.98` (+12 dB). For fades and ducking use the `data-automation` volume lane (`hyperframes-core` skill, `creator-editing-recipes.md`)
14+
- `data-volume="0.8"` — Audio/video gain. `1` is 0 dB, `0` is silence, and values above `1` boost up to `3.98` (+12 dB). For ducking and shaped envelopes use the `data-automation` volume lane (`hyperframes-core` skill, `creator-editing-recipes.md`)
15+
- `data-fade-in="0.5"` / `data-fade-out="1"` — Clip-edge fades (seconds): linear gain ramps anchored to the clip's start and end, multiplied on top of `data-volume` and any volume lane
1516
- `data-has-audio="true"` — Indicates video has an audio track
1617

1718
## Composition

‎packages/core/docs/core.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,7 @@ Background music or sound effects. Audio clips are invisible.
283283
```
284284

285285
- `data-media-start` — Playback begins at this time in the source audio file (seconds). Default: `0`.
286+
- `data-fade-in` / `data-fade-out` — (optional) Clip-edge fades in seconds. Gain ramps linearly from silence over the first `data-fade-in` seconds and back to silence over the last `data-fade-out` seconds of the clip, on top of `data-volume` and any `data-automation` volume lane. Anchored to the clip's edges, so a fade-out follows a later trim. Also honoured on `<video>` audio.
286287
- `data-duration` — (optional) How long the clip occupies on the timeline, in seconds. Playback runs from `data-media-start` for up to `data-duration` seconds. If the source media runs out before `data-duration` elapses, playback naturally stops (the clip remains mounted but silent). If omitted, defaults to the remaining duration of the source file from `data-media-start`.
287288

288289
## Two Layers: Primitives and Scripts

‎packages/core/package-subpaths.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,12 @@
188188
"types": "./dist/audioGain.d.ts",
189189
"environments": ["browser", "bun", "node"]
190190
},
191+
"./audio-fade": {
192+
"source": "./src/audioFade.ts",
193+
"runtime": "./dist/audioFade.js",
194+
"types": "./dist/audioFade.d.ts",
195+
"environments": ["browser", "bun", "node"]
196+
},
191197
"./speed-ramp": {
192198
"source": "./src/speedRamp.ts",
193199
"runtime": "./dist/speedRamp.js",

‎packages/core/package.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,12 @@
202202
"import": "./src/audioGain.ts",
203203
"types": "./src/audioGain.ts"
204204
},
205+
"./audio-fade": {
206+
"bun": "./src/audioFade.ts",
207+
"node": "./dist/audioFade.js",
208+
"import": "./src/audioFade.ts",
209+
"types": "./src/audioFade.ts"
210+
},
205211
"./speed-ramp": {
206212
"bun": "./src/speedRamp.ts",
207213
"node": "./dist/speedRamp.js",
@@ -546,6 +552,10 @@
546552
"import": "./dist/audioGain.js",
547553
"types": "./dist/audioGain.d.ts"
548554
},
555+
"./audio-fade": {
556+
"import": "./dist/audioFade.js",
557+
"types": "./dist/audioFade.d.ts"
558+
},
549559
"./speed-ramp": {
550560
"import": "./dist/speedRamp.js",
551561
"types": "./dist/speedRamp.d.ts"
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
import { describe, expect, it } from "vitest";
2+
import {
3+
clampFadesToDuration,
4+
fadeGain,
5+
formatFadeSeconds,
6+
readElementFades,
7+
readFadeSeconds,
8+
} from "./audioFade.js";
9+
10+
describe("readFadeSeconds", () => {
11+
it("reads a positive number and treats everything else as no fade", () => {
12+
expect(readFadeSeconds("0.5")).toBe(0.5);
13+
expect(readFadeSeconds("2")).toBe(2);
14+
expect(readFadeSeconds(null)).toBe(0);
15+
expect(readFadeSeconds("")).toBe(0);
16+
expect(readFadeSeconds("-1")).toBe(0);
17+
expect(readFadeSeconds("NaN")).toBe(0);
18+
expect(readFadeSeconds("abc")).toBe(0);
19+
});
20+
21+
it("reads both attributes off an element", () => {
22+
const attrs: Record<string, string> = { "data-fade-in": "0.25", "data-fade-out": "1.5" };
23+
expect(readElementFades({ getAttribute: (n) => attrs[n] ?? null })).toEqual({
24+
fadeIn: 0.25,
25+
fadeOut: 1.5,
26+
});
27+
expect(readElementFades({ getAttribute: () => null })).toEqual({ fadeIn: 0, fadeOut: 0 });
28+
});
29+
});
30+
31+
describe("clampFadesToDuration", () => {
32+
it("leaves fades that fit alone", () => {
33+
expect(clampFadesToDuration({ fadeIn: 1, fadeOut: 2 }, 10)).toEqual({ fadeIn: 1, fadeOut: 2 });
34+
});
35+
it("scales fades that outrun the clip so they meet inside it", () => {
36+
expect(clampFadesToDuration({ fadeIn: 6, fadeOut: 6 }, 6)).toEqual({ fadeIn: 3, fadeOut: 3 });
37+
expect(clampFadesToDuration({ fadeIn: 3, fadeOut: 1 }, 2)).toEqual({
38+
fadeIn: 1.5,
39+
fadeOut: 0.5,
40+
});
41+
});
42+
it("keeps authored fades on a clip of unknown duration", () => {
43+
expect(clampFadesToDuration({ fadeIn: 4, fadeOut: 4 }, Number.POSITIVE_INFINITY)).toEqual({
44+
fadeIn: 4,
45+
fadeOut: 4,
46+
});
47+
});
48+
});
49+
50+
describe("fadeGain", () => {
51+
const fades = { fadeIn: 2, fadeOut: 1 };
52+
it("is unity with no fades", () => {
53+
expect(fadeGain(0, 10, { fadeIn: 0, fadeOut: 0 })).toBe(1);
54+
expect(fadeGain(10, 10, { fadeIn: 0, fadeOut: 0 })).toBe(1);
55+
});
56+
it("ramps linearly from silence over the fade-in", () => {
57+
expect(fadeGain(0, 10, fades)).toBe(0);
58+
expect(fadeGain(1, 10, fades)).toBe(0.5);
59+
expect(fadeGain(2, 10, fades)).toBe(1);
60+
expect(fadeGain(5, 10, fades)).toBe(1);
61+
});
62+
it("ramps linearly to silence over the fade-out, anchored at the clip end", () => {
63+
expect(fadeGain(9, 10, fades)).toBe(1);
64+
expect(fadeGain(9.5, 10, fades)).toBe(0.5);
65+
expect(fadeGain(10, 10, fades)).toBe(0);
66+
expect(fadeGain(11, 10, fades)).toBe(0);
67+
});
68+
it("multiplies where the two fades overlap on a short clip", () => {
69+
// 2 s clip, both fades scaled to 1 s each: midpoint is the seam.
70+
expect(fadeGain(1, 2, { fadeIn: 2, fadeOut: 2 })).toBe(1);
71+
expect(fadeGain(0.5, 2, { fadeIn: 2, fadeOut: 2 })).toBe(0.5);
72+
});
73+
it("never applies a fade-out to a clip of unknown duration", () => {
74+
expect(fadeGain(100, Number.POSITIVE_INFINITY, fades)).toBe(1);
75+
expect(fadeGain(1, Number.POSITIVE_INFINITY, fades)).toBe(0.5);
76+
});
77+
it("treats 0, negative, and NaN fades as no fade", () => {
78+
expect(fadeGain(0, 10, { fadeIn: 0, fadeOut: 0 })).toBe(1);
79+
expect(fadeGain(0, 10, { fadeIn: -1, fadeOut: -4 })).toBe(1);
80+
expect(fadeGain(0, 10, { fadeIn: Number.NaN, fadeOut: Number.NaN })).toBe(1);
81+
});
82+
it("returns a finite 0..1 gain for NaN elapsed and for fades longer than the clip", () => {
83+
expect(fadeGain(Number.NaN, 10, fades)).toBe(1);
84+
const overlapping = fadeGain(0.5, 2, { fadeIn: 10, fadeOut: 10 });
85+
expect(overlapping).toBeGreaterThanOrEqual(0);
86+
expect(overlapping).toBeLessThanOrEqual(1);
87+
expect(Number.isFinite(overlapping)).toBe(true);
88+
});
89+
});
90+
91+
describe("formatFadeSeconds", () => {
92+
it("writes short attribute text", () => {
93+
expect(formatFadeSeconds(0)).toBe("0");
94+
expect(formatFadeSeconds(1)).toBe("1");
95+
expect(formatFadeSeconds(0.5)).toBe("0.5");
96+
expect(formatFadeSeconds(0.333)).toBe("0.33");
97+
expect(formatFadeSeconds(-2)).toBe("0");
98+
});
99+
});

‎packages/core/src/audioFade.ts‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
/** Clip-edge fades: `data-fade-in` / `data-fade-out` in seconds, linear, multiplied on the resolved level. */
2+
3+
export const HF_AUDIO_FADE_IN_ATTR = "data-fade-in";
4+
export const HF_AUDIO_FADE_OUT_ATTR = "data-fade-out";
5+
6+
export const HF_AUDIO_FADE_IN_DATA_KEY = HF_AUDIO_FADE_IN_ATTR.slice("data-".length);
7+
export const HF_AUDIO_FADE_OUT_DATA_KEY = HF_AUDIO_FADE_OUT_ATTR.slice("data-".length);
8+
9+
export interface AudioFades {
10+
/** Seconds from the clip start over which gain rises 0 → 1. */
11+
fadeIn: number;
12+
/** Seconds before the clip end over which gain falls 1 → 0. */
13+
fadeOut: number;
14+
}
15+
16+
export const NO_FADES: Readonly<AudioFades> = Object.freeze({ fadeIn: 0, fadeOut: 0 });
17+
18+
function finiteFadeSeconds(value: number): number {
19+
return Number.isFinite(value) && value > 0 ? value : 0;
20+
}
21+
22+
/** Attribute text as seconds: finite and positive, else 0 (no fade). */
23+
export function readFadeSeconds(raw: string | null | undefined): number {
24+
if (raw == null || raw === "") return 0;
25+
return finiteFadeSeconds(Number.parseFloat(raw));
26+
}
27+
28+
export function readElementFades(el: { getAttribute(name: string): string | null }): AudioFades {
29+
return {
30+
fadeIn: readFadeSeconds(el.getAttribute(HF_AUDIO_FADE_IN_ATTR)),
31+
fadeOut: readFadeSeconds(el.getAttribute(HF_AUDIO_FADE_OUT_ATTR)),
32+
};
33+
}
34+
35+
/** Scale fades that outrun the clip so they meet inside it. Infinite duration keeps them as authored. */
36+
export function clampFadesToDuration(fades: AudioFades, duration: number): AudioFades {
37+
const fadeIn = finiteFadeSeconds(fades.fadeIn);
38+
const fadeOut = finiteFadeSeconds(fades.fadeOut);
39+
if (!Number.isFinite(duration) || duration <= 0) return { fadeIn, fadeOut };
40+
const total = fadeIn + fadeOut;
41+
if (total <= duration) return { fadeIn, fadeOut };
42+
const scale = duration / total;
43+
return { fadeIn: fadeIn * scale, fadeOut: fadeOut * scale };
44+
}
45+
46+
/** Gain at `elapsed` into a clip of `duration`. 1 where no fade runs; the two multiply where they meet. */
47+
export function fadeGain(elapsed: number, duration: number, fades: AudioFades): number {
48+
if (!Number.isFinite(elapsed)) return 1;
49+
const { fadeIn, fadeOut } = clampFadesToDuration(fades, duration);
50+
let gain = 1;
51+
if (fadeIn > 0) gain *= clamp01(elapsed / fadeIn);
52+
if (fadeOut > 0 && Number.isFinite(duration)) gain *= clamp01((duration - elapsed) / fadeOut);
53+
return gain;
54+
}
55+
56+
/** Attribute text Studio writes: up to 2 decimals, no trailing zeros. */
57+
export function formatFadeSeconds(seconds: number): string {
58+
const rounded = Math.round(finiteFadeSeconds(seconds) * 100) / 100;
59+
return Number.isInteger(rounded) ? `${rounded}` : rounded.toFixed(2).replace(/0+$/, "");
60+
}
61+
62+
function clamp01(value: number): number {
63+
if (!Number.isFinite(value) || value < 0) return 0;
64+
return value > 1 ? 1 : value;
65+
}

‎packages/core/src/audioGain.test.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,22 @@ import {
44
formatAudioGain,
55
AUDIO_GAIN_FADER_MIN,
66
MAX_AUDIO_GAIN,
7+
audioDbToGain,
8+
audioGainToDb,
79
audioGainToFaderPosition,
810
audioGainToText,
911
audioFaderPositionToGain,
1012
} from "./audioGain";
1113

1214
describe("audio gain fader", () => {
15+
it("converts decibels to linear gain", () => {
16+
expect(audioDbToGain(0)).toBe(1);
17+
expect(audioDbToGain(-6)).toBeCloseTo(10 ** (-6 / 20), 10);
18+
expect(MAX_AUDIO_GAIN).toBe(audioDbToGain(12));
19+
expect(audioGainToDb(1)).toBeCloseTo(0, 10);
20+
expect(audioGainToDb(audioDbToGain(-6))).toBeCloseTo(-6, 10);
21+
});
22+
1323
it("puts unity gain at the physical midpoint", () => {
1424
expect(audioGainToFaderPosition(1)).toBe(0);
1525
expect(audioFaderPositionToGain(0)).toBe(1);

‎packages/core/src/audioGain.ts‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,15 @@
66
* shared ceiling here so Studio, preview, and render cannot drift.
77
*/
88
export const MAX_AUDIO_GAIN_DB = 12;
9-
export const MAX_AUDIO_GAIN = 10 ** (MAX_AUDIO_GAIN_DB / 20);
9+
/** Linear gain from a decibel value. Owner of the dB → gain formula. */
10+
export function audioDbToGain(db: number): number {
11+
return 10 ** (db / 20);
12+
}
13+
/** Inverse of `audioDbToGain`. Caller must pass a positive finite gain. */
14+
export function audioGainToDb(gain: number): number {
15+
return 20 * Math.log10(gain);
16+
}
17+
export const MAX_AUDIO_GAIN = audioDbToGain(MAX_AUDIO_GAIN_DB);
1018

1119
/** Studio fader coordinates. Unity is deliberately the physical midpoint. */
1220
export const AUDIO_GAIN_FADER_MIN = -100;
@@ -87,13 +95,13 @@ export function audioFaderPositionToGain(position: number): number {
8795
safe < 0
8896
? (safe / Math.abs(AUDIO_GAIN_FADER_MIN)) * Math.abs(MIN_AUDIO_GAIN_DB)
8997
: (safe / AUDIO_GAIN_FADER_MAX) * MAX_AUDIO_GAIN_DB;
90-
return 10 ** (db / 20);
98+
return audioDbToGain(db);
9199
}
92100

93101
export function audioGainToFaderPosition(gain: number): number {
94102
const safe = clampAudioGain(gain);
95103
if (safe === 0) return AUDIO_GAIN_FADER_MIN;
96-
const db = 20 * Math.log10(safe);
104+
const db = audioGainToDb(safe);
97105
const position =
98106
db < 0
99107
? (db / Math.abs(MIN_AUDIO_GAIN_DB)) * Math.abs(AUDIO_GAIN_FADER_MIN)
@@ -104,7 +112,7 @@ export function audioGainToFaderPosition(gain: number): number {
104112
export function audioGainToText(gain: number): string {
105113
const safe = clampAudioGain(gain);
106114
if (safe === 0) return "-∞ dB";
107-
const db = 20 * Math.log10(safe);
115+
const db = audioGainToDb(safe);
108116
const rounded = Math.abs(db) < 0.05 ? 0 : db;
109117
return (rounded > 0 ? "+" : "") + rounded.toFixed(1) + " dB";
110118
}

‎packages/core/src/runtime/media.test.ts‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,6 +465,56 @@ describe("syncRuntimeMedia", () => {
465465
expect(only).toBeCloseTo(0.55, 5);
466466
});
467467

468+
it("applies data-fade-in / data-fade-out on top of data-volume, anchored to the clip edges", () => {
469+
const at = (t: number) => {
470+
const clip = createMockClip({ start: 2, end: 12, duration: 10, volume: 0.8 });
471+
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
472+
clip.el.setAttribute("data-fade-in", "2");
473+
clip.el.setAttribute("data-fade-out", "1");
474+
clip.fades = { fadeIn: 2, fadeOut: 1 };
475+
let authored = -1;
476+
syncRuntimeMedia({
477+
clips: [clip],
478+
timeSeconds: t,
479+
playing: true,
480+
playbackRate: 1,
481+
onElementVolume: (_el, _effective, authorVolume) => {
482+
authored = authorVolume;
483+
},
484+
});
485+
return authored;
486+
};
487+
expect(at(2.5)).toBeCloseTo(0.2, 5); // a quarter into the 2 s fade-in
488+
expect(at(3)).toBeCloseTo(0.4, 5); // halfway through the fade-in
489+
expect(at(6)).toBeCloseTo(0.8, 5); // body of the clip: data-volume alone
490+
expect(at(11.5)).toBeCloseTo(0.4, 5); // halfway through the 1 s fade-out
491+
expect(at(11.9)).toBeCloseTo(0.08, 5); // almost at the clip's end
492+
});
493+
494+
it("never writes NaN or a negative volume for 0, negative, NaN, or longer-than-clip fades", () => {
495+
const cases: Array<{ fadeIn: number; fadeOut: number }> = [
496+
{ fadeIn: 0, fadeOut: 0 },
497+
{ fadeIn: -2, fadeOut: -1 },
498+
{ fadeIn: Number.NaN, fadeOut: Number.NaN },
499+
{ fadeIn: 40, fadeOut: 40 },
500+
];
501+
for (const fades of cases) {
502+
const clip = createMockClip({ start: 0, end: 10, duration: 10, volume: 0.5, fades });
503+
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
504+
for (const t of [0, 5, 10, 12]) {
505+
syncRuntimeMedia({
506+
clips: [clip],
507+
timeSeconds: t,
508+
playing: true,
509+
playbackRate: 1,
510+
});
511+
expect(Number.isFinite(clip.el.volume)).toBe(true);
512+
expect(clip.el.volume).toBeGreaterThanOrEqual(0);
513+
expect(clip.el.volume).toBeLessThanOrEqual(1);
514+
}
515+
}
516+
});
517+
468518
it("sends boosted author gain to Web Audio while keeping the native element legal", () => {
469519
const clip = createMockClip({ start: 0, end: 10, volume: 3.98 });
470520
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });

0 commit comments

Comments
 (0)