fix: FIT-31: Ensure media buffers are syncable events that keep audio and video in sync #7633

bmartel · 2025-05-27T15:18:45Z

Previously we were not syncing when a media source would buffer, which led to out of sync audio and video in many cases, especially in spotty network conditions or with very large file sizes. This addresses that issue by syncing buffering state events into SyncManager, and handling the state in the particular cases Audio, Video, Paragraphs. Also updated and aligned the buffering visuals by moving that into the TimelineControls.

This video showcases a ~300MB video throttled to 3G then seeked and played, and later unthrottled to simulate what was previously causing media sources to go out of sync with one another:

Kapture.2025-05-28.at.08.47.55.webm

Feature Flag:

Added a new feature flag FF_SYNCED_BUFFERING to enable synchronized media buffering across components. (web/libs/core/src/lib/utils/feature-flags/flags.ts)

UI Updates:

Introduced a visual buffering indicator in the Timeline component's controls, including new SCSS styles and animations. (web/libs/editor/src/components/Timeline/Controls.scss) [1] [2]
Updated the VideoCanvas component to conditionally display a buffering indicator based on the buffering state. (web/libs/editor/src/components/VideoCanvas/VideoCanvas.scss, web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx) [1] [2]

Synchronized Buffering Logic:

Added a useSyncedBuffering hook to manage synchronized buffering states across components using Jotai atoms. (web/libs/editor/src/hooks/useSyncedBuffering.ts)
Updated the VideoCanvas and Html5Player components to utilize the synchronized buffering logic, including debounced state updates and conditional behavior based on the feature flag. (web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx, web/libs/editor/src/lib/AudioUltra/Controls/Html5Player.ts) [1] [2]

Playback Logic Enhancements:

Modified the Player class to include a buffering property and handle buffering events (canplay and waiting) with debounced updates. (web/libs/editor/src/lib/AudioUltra/Controls/Player.ts) [1] [2]
Adjusted the Timeline and VideoCanvas components to propagate buffering states through their props and interfaces. (web/libs/editor/src/components/Timeline/Types.ts, web/libs/editor/src/components/Timeline/Timeline.tsx) [1] [2]

These changes collectively improve the user experience by providing better visual feedback during buffering and ensuring consistent playback behavior across components.

… and video in sync

…ntrol spacing on video

netlify · 2025-05-27T15:18:50Z

✅ Deploy Preview for heartex-docs canceled.

Name	Link
🔨 Latest commit	`4a4ed26`
🔍 Latest deploy log	https://app.netlify.com/projects/heartex-docs/deploys/68a3581079cc9900084e32b8

netlify · 2025-05-27T15:18:52Z

✅ Deploy Preview for label-studio-storybook ready!

Name	Link
🔨 Latest commit	`4a4ed26`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-storybook/deploys/68a358101514690008796690
😎 Deploy Preview	https://deploy-preview-7633--label-studio-storybook.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify · 2025-05-27T15:19:19Z

✅ Deploy Preview for label-studio-playground ready!

Name	Link
🔨 Latest commit	`4a4ed26`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-playground/deploys/68a35810a8ed590008883dfd
😎 Deploy Preview	https://deploy-preview-7633--label-studio-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify · 2025-05-27T15:19:20Z

✅ Deploy Preview for label-studio-docs-new-theme canceled.

Name	Link
🔨 Latest commit	`4a4ed26`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-docs-new-theme/deploys/68a3581027162300080260fb

bmartel · 2025-05-28T13:44:07Z

/fm sync

Workflow run

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15301787592

…would change

Gondragos · 2025-07-14T13:18:49Z

/git merge

Workflow run
Successfully merged: create mode 100644 web/libs/editor/src/components/TaskSummary/tests/TaskSummary.test.tsx

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16267970518

web/libs/editor/src/components/Timeline/Controls.scss

# Conflicts: # web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx # web/libs/editor/src/tags/object/Video/HtxVideo.jsx # web/libs/frontend-test/src/helpers/LSF/VideoView.ts

Co-authored-by: Andrew <[email protected]>

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16600887656

# Conflicts: # web/libs/frontend-test/src/helpers/LSF/VideoView.ts

# Conflicts: # web/libs/editor/src/tags/object/Paragraphs/HtxParagraphs.jsx

niklub · 2025-08-05T10:32:00Z

/git merge

Workflow run
Successfully merged: delete mode 100644 web/libs/app-common/src/blocks/StorageProviderForm/providers/localFiles.ts

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16747575052

niklub · 2025-08-10T12:34:49Z

/git merge

Workflow run
Successfully merged: create mode 100644 label_studio/tests/data_import/test_streaming_import.py

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16861459102

niklub · 2025-08-11T14:28:16Z

/git merge

Workflow run
Successfully merged: create mode 100644 web/libs/editor/src/lib/AudioUltra/Visual/Renderer/ResizeRenderer.ts

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16883035943

niklub · 2025-08-14T11:19:08Z

/git merge

Workflow run
Successfully merged: delete mode 100644 web/libs/editor/src/regions/ImageRegion.js

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16963704331

Gondragos · 2025-08-18T12:31:08Z

/git merge

Workflow run
Successfully merged: 79 files changed, 1663 insertions(+), 347 deletions(-)

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/17040659999

Fix for: Failed to set the 'currentTime' property on 'HTMLMediaElement': The provided double value is non-finite

Gondragos · 2025-08-18T14:53:06Z

/git merge

Workflow run
Successfully merged: 7 files changed, 159 insertions(+), 20 deletions(-)

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/17044174008

bmartel added 6 commits May 26, 2025 16:42

fix: FIT-31: Ensure media buffers are syncable events that keep audio…

731fab2

… and video in sync

put all changes behind FF

eb01cba

fixing FF usage, aligning buffer styling with all media types, fix co…

cd9a9e4

…ntrol spacing on video

tweaking the media buffering animations

55b67f0

tweaking the media buffering animations

c427c49

update Audio loader and error colors to use design tokens

6184cd5

bmartel requested review from a team, Gondragos, hlomzik and nick-skriabin as code owners May 27, 2025 15:18

github-actions bot added the fix label May 27, 2025

bmartel added 2 commits May 27, 2025 11:54

debounce updates to smooth out the handling of buffering

e437ef4

Merge remote-tracking branch 'origin/develop' into fb-fit-31

bcc319f

bmartel added 2 commits May 28, 2025 13:47

Sync Follow Merge dependencies

a9b22e5

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15301787592

Merge branch 'develop' into 'fb-fit-31'

502df27

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15301787592

robot-ci-heartex added the FM Stale label May 28, 2025

bmartel changed the title ~~fix: FIT-31: Ensure media buffers are syncable events that keep audio… … and video in sync~~ fix: FIT-31: Ensure media buffers are syncable events that keep audio and video in sync May 29, 2025

bmartel added 8 commits May 29, 2025 08:40

video should not progress until buffering is over

0719edd

Merge remote-tracking branch 'origin/develop' into fb-fit-31

9f431b8

use synced buffering

5d7d248

don't over report buffering, only report when all buffering elements …

2f3ba48

…would change

Merge remote-tracking branch 'origin/develop' into fb-fit-31

7756fbf

cleanup and waveform buffer sync to player instance

50a7d02

fix syncing A/V + Paragraphs

c7e3fd5

don't allow video to seek while buffering it causes a loop

a06a9c4

fix: Handle browser closure errors in error collector

a12cb8d

Gondragos and others added 2 commits July 14, 2025 13:19

Merge branch 'develop' into 'fb-fit-31'

d5fb34b

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16267970518

Fix phrases playing display, optimize rendering, increase stability

999ac2b

hlomzik reviewed Jul 24, 2025

View reviewed changes

web/libs/editor/src/components/Timeline/Controls.scss Outdated Show resolved Hide resolved

Gondragos and others added 6 commits July 25, 2025 00:49

fix: Simplify draw handling in Visualizer

c57f9a0

Merge branch 'develop' into fb-fit-31

e2e5a96

# Conflicts: # web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx # web/libs/editor/src/tags/object/Video/HtxVideo.jsx # web/libs/frontend-test/src/helpers/LSF/VideoView.ts

Accept suggestions

8f6015e

Co-authored-by: Andrew <[email protected]>

ci: Build tag docs

edf1d9f

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16600887656

Merge branch 'develop' into fb-fit-31

05ec5bf

# Conflicts: # web/libs/frontend-test/src/helpers/LSF/VideoView.ts

Merge branch 'develop' into fb-fit-31

0269740

# Conflicts: # web/libs/editor/src/tags/object/Paragraphs/HtxParagraphs.jsx

niklub and others added 2 commits August 5, 2025 10:32

Merge branch 'develop' into 'fb-fit-31'

27f8cdb

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16747575052

Merge branch 'develop' into fb-fit-31

7903945

Merge branch 'develop' into 'fb-fit-31'

635c635

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16861459102

Merge branch 'develop' into 'fb-fit-31'

5bd4d35

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16883035943

Merge branch 'develop' into 'fb-fit-31'

e66999b

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/16963704331

Gondragos and others added 4 commits August 18, 2025 12:31

Merge branch 'develop' into 'fb-fit-31'

6ee9f51

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/17040659999

fix: Make Sync buffering playback test more stable

8321855

Fix types

3ff1760

Fix pausing before load

3dfdc6b

Fix for: Failed to set the 'currentTime' property on 'HTMLMediaElement': The provided double value is non-finite

Gondragos and others added 2 commits August 18, 2025 14:53

Merge branch 'develop' into 'fb-fit-31'

53e5a72

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/17044174008

Fix auto merge consequences

4a4ed26

niklub merged commit 0fd582d into develop Aug 21, 2025
63 of 66 checks passed

robot-ci-heartex deleted the fb-fit-31 branch August 21, 2025 09:56

fix: FIT-31: Ensure media buffers are syncable events that keep audio and video in sync #7633

fix: FIT-31: Ensure media buffers are syncable events that keep audio and video in sync #7633

Uh oh!

Conversation

bmartel commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Feature Flag:

UI Updates:

Synchronized Buffering Logic:

Playback Logic Enhancements:

Uh oh!

netlify bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for heartex-docs canceled.

Uh oh!

netlify bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for label-studio-storybook ready!

Uh oh!

netlify bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for label-studio-playground ready!

Uh oh!

netlify bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for label-studio-docs-new-theme canceled.

Uh oh!

bmartel commented May 28, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gondragos commented Jul 14, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

niklub commented Aug 5, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niklub commented Aug 10, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niklub commented Aug 11, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niklub commented Aug 14, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gondragos commented Aug 18, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gondragos commented Aug 18, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bmartel commented May 27, 2025 •

edited

Loading

netlify bot commented May 27, 2025 •

edited

Loading

netlify bot commented May 27, 2025 •

edited

Loading

netlify bot commented May 27, 2025 •

edited

Loading

netlify bot commented May 27, 2025 •

edited

Loading

bmartel commented May 28, 2025 •

edited by robot-ci-heartex

Loading

Gondragos commented Jul 14, 2025 •

edited by robot-ci-heartex

Loading

niklub commented Aug 5, 2025 •

edited by robot-ci-heartex

Loading

niklub commented Aug 10, 2025 •

edited by robot-ci-heartex

Loading

niklub commented Aug 11, 2025 •

edited by robot-ci-heartex

Loading

niklub commented Aug 14, 2025 •

edited by robot-ci-heartex

Loading

Gondragos commented Aug 18, 2025 •

edited by robot-ci-heartex

Loading

Gondragos commented Aug 18, 2025 •

edited by robot-ci-heartex

Loading