Skip to content

Instantly share code, notes, and snippets.

@robinebers
Created October 28, 2026 06:01
Show Gist options
  • Select an option

  • Save robinebers/d06818f0c2fae0f15e11b6decd1ae529 to your computer and use it in GitHub Desktop.

Select an option

Save robinebers/d06818f0c2fae0f15e11b6decd1ae529 to your computer and use it in GitHub Desktop.

HyperFrames preview on iPhone: what we measured

From Attention Machine (Robin Ebers). This is the iPhone part of our report on 0.8.210. On desktop Chrome, 0.8.210 passes everything in our first report. On an iPhone, two things go wrong. We reproduced both on a real iPhone and in the iOS Simulator, with a trace for every play.

Setup

Device iPhone 18 Pro Max (iPhone19,7), iOS 27.0.1, Safari 27.0.1, connected by USB
Also iOS Simulator: iPhone 17 Pro, iOS 26.3 (Xcode 26.3). Same results on every measure below except where noted.
Build @hyperframes/player + @hyperframes/core 0.8.210
Page https://media-dev.amcdn.app/hyperframes-demo/0.8.210.app-c8c4dd444/index.html (?case=primary = 3 takes, ?case=tentake = 10 takes)
Composition Same as our first report: one muted <video> and one <audio> per take, all on one MP4. Page and media are on the same origin, served by Cloudflare over HTTP/2.
How we drove it safaridriver on the Mac drove Safari on the phone. One touch on a Play button in the host page calls player.play(), the same as a person tapping play. Before the tap we wrapped HTMLMediaElement (play() and its result, seeks, rate, volume, muted), AudioContext methods, requestVideoFrameCallback on every video, and getVideoPlaybackQuality. We also sampled every element and __player.getTime() once per frame.
Runs 3 plays of each edit on 0.8.210 as is. 3 plays of each edit with the fix below added. 3 plays of each edit with a single <audio>. Simulator: 3 plays of each edit, plus two small test pages.

The table under the player in our demo only watches the <video> elements, so it showed nothing wrong on the iPhone. Both problems are in the audio.

Problem 1: after about 5 seconds, every take that starts is silent for the rest of the edit

What a user sees. The edit plays with sound at first. From the first take that starts more than about 5 s after pressing play, there is no sound at all until the end. The picture keeps playing. On our 3-take edit the last 4 s are silent. On our 10-take edit only the first take is heard, so 47 of 53 s are silent.

What we measured (iPhone, 6 of 6 plays; Simulator, 6 of 6).

Edit First play() refused Time since the tap Sound stops at (edit time) Refused play() calls per play
3 takes take 3's <audio> 21.98–22.03 s 20.66–20.67 s of 24.8 s 135–136
10 takes take 2's <audio> 6.95–6.99 s 6.25–6.28 s of 53.08 s 1,475–1,483

Cause. In iOS Safari an unmuted media element may only start while the page still has transient user activation, which lasts about 5 s after the tap. A separate Simulator test page started fresh <audio> elements at fixed delays after one tap. They played at 0–4.5 s, and every play() from 5.0 s on was refused with NotAllowedError. On the iPhone, play() worked at 3.5 s and was refused at 6.95 s and later.

HyperFrames starts each take's <audio> when that take's audio window opens (packages/core/src/runtime/media.ts:707). Any take that starts after the window gets NotAllowedError. From there:

  1. media.ts:720 treats it as autoplay blocked, init.ts:3669 posts media-autoplay-blocked once, and the player calls promoteToParentProxy (packages/player/src/runtime-message-handler.ts:136, parent-media.ts:311).
  2. parent-media.ts:321 mutes every <video>/<audio> in the composition. That includes the take that was still sounding.
  3. The parent-frame Audio() copies were created at ready and have never been played inside a tap. Their play() is refused too. _gateEntryPlayback (parent-media.ts:214) retries it on every mirror tick, about 30 times a second. _reportPlaybackError (:392) fires one playbackerror and then stays quiet.
  4. Nothing hands audio back. _audioOwner only returns to "runtime" in resetForIframeLoad/destroy (:98, :123).

So the fallback added for blocked autoplay (#6417) is what makes it silent: it mutes the composition and moves the sound to elements that iOS refuses as well. The desktop simulation in our first report could not show this, because desktop Chrome lets any element play once the page has been tapped.

Fix we tested on the iPhone. Inside the tap, call play() and then pause() once on every take's <audio>. iOS then allows each of those elements to start later without a tap. Our Simulator test page showed this per element: an element played and paused in the tap started fine 20 s later, and an element never played in the tap was refused. On the real iPhone with this added (3 + 3 plays): 0 refused play() calls, and sound to the end of both edits. In HyperFrames this would mean the player's play() (which reaches the composition by postMessage, hyperframes-player.ts:442, still inside the activation window) primes every audio clip. The parent copies need the same, or they should not be used when the composition can still play.

Repro. Open the demo URL on an iPhone, tap play on the 10-take edit, and listen. Sound stops at 0:06. In Web Inspector, play() on the second <audio> rejects with NotAllowedError and the player fires audioownershipchange {owner:"parent", reason:"autoplay-blocked"}.

Problem 2: the picture stops at the start and at every cut, because the playhead waits for each new <audio>

What a user sees. A stutter right after pressing play, and a stop of about half a second at the cuts. Sometimes it is longer than a second. It is easiest to see once Problem 1 is fixed, because then every cut has working audio.

What we measured (iPhone).

Runs (3 plays each) Playhead stops over 50 ms Longest stop Picture gaps over 100 ms Edit ran longer by
3 takes, as is 6 per play: 3 in the first 0.5 s, 3 at cut 1 331–394 ms 3–4 (max 351 ms) 1.34–1.37 s
10 takes, as is (silent after 6.3 s) 3 per play, all in the first 0.7 s 317–366 ms 2–3 0.70–0.74 s
3 takes, with the fix from Problem 1 7–8 per play 352–405 ms 5–6 1.93–2.04 s
10 takes, with the fix from Problem 1 31–36 per play: about 3 at every cut 533–1,533 ms 28–31 (max 1,853 ms) 8.24–8.86 s

The video decoder is not the cause: getVideoPlaybackQuality reported 0–2 dropped frames out of 677–1,490 per play. The 10-take edit runs smoothly after 6.3 s in the "as is" row only because its audio is dead by then, and the clock falls back to a monotonic timer in proxy mode (init.ts:5531).

Cause, from one cut (10 takes, with the fix, cut at 6.4 s). Times are wall time after the tap.

  1. 7.24 s: take 2's <audio> gets play() and fires playing. Its currentTime then stays at 4.280 for about 300 ms. That is how long iOS takes before a newly started element's time moves.
  2. 7.54 s, playhead 6.575: take 1's audio window ends and is paused. The clock moves to take 2's element (refreshTransportClockSource, init.ts:5543), which is now about 300 ms behind. steerMedia seeks it, because the lag is over AUDIO_STEER_LIMIT_SECONDS = 0.1 (mediaSteer.ts:14, :30). That jump is audible.
  3. While it seeks, clock.ts:93 holds the playhead (if (el.seeking) return this._lastNow). The <video> gets ahead and is paused and played again, so the picture freezes for about 250 ms.
  4. 8.19 s: after the seek the element's time freezes again for about 280 ms while iOS restarts output. clock.ts:102 holds the playhead again, and the video pauses again.

The same pattern of three stops shows at all 9 cuts, and once at the start (take 1's <audio> not moving yet).

Possible fix. Don't hand the clock to an <audio> whose currentTime has not started moving yet. Or account for the iOS start-up delay (start the incoming element earlier, as the overlap already allows). Or drive the clock from AudioContext.currentTime for routed elements, which keeps moving. Avoid seeking an element that is sounding for a lag that comes from start-up and not from drift.

Supporting test (iPhone). With all takes' audio collapsed into one <audio> for the whole edit (in-point of take 1, full length, other <audio> removed), there were 0 refused play() calls and no stops at any cut (3 + 3 plays). Only the start-up stops in the first second were left (0.53–1.01 s in total). This tests the element layout, not the sound: the audio content was wrong on purpose.

Checked on the iPhone and not a problem

  • Preload. All 9 (3 takes) and 30 (10 takes) media elements reached readyState 4 before play, parent copies included. preload="auto" is honoured.
  • Web Audio. One AudioContext, running the whole time, never interrupted or suspended. 0 decodeAudioData calls, so our 76 MB file is never decoded whole.
  • Playback rate. 0 of the rate writes were refused or clamped. Every value read back as written.
  • Network. Media requests are HTTP/2 206 with correct Content-Range, served from the Cloudflare cache. 24–98 requests per play. No stalls tied to requests.
  • Dropped frames. 0–2 per play.

On our side

  • Our test table is blind on iPhone. It scores only <video>. We will add the audio checks used here.
  • One audio file per edit. Our edits use one <audio> per take on a shared file. We will render one audio-only file per edit, with the joins and gains baked in, and use a single <audio> clip. On the iPhone that layout avoided both problems (see the supporting test above). We'd still like Problem 1 fixed in the player, because any composition with more than one audio clip hits it.

Evidence

Per-play traces (trace.json, every media event, play() result and per-frame sample), result tables, and the harness that drove the phone are in our evidence folder (renders/ATTE-162/). We can send them on request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment