Repository navigation
Description
| Bad | Expected |
|---|---|
tracking-bad.mov |
tracking-expected.mov |
When touch event is received, instead of immediately beginning a frame Flutter waits for next vsync. But that skips an entire frame. By the time vsync comes, the frame should be already rendered and ready to go. But instead it has just began. On iOS touch events seem to be delivered within few milliseconds of the display link callback so there should be enough time.
You can pause the "bad" video to see that during scrolling Flutter lags behind native by one frame exactly.
The expected video is with quick and dirty engine modification that forces BeginFrame immediately on input event.
The way vsync is implemented in Flutter might need some work. It should make sure that repaints are synchronized with display updates, but it shouldn't be causing additional latency.
Activity
knopp commentedon Aug 28, 2022
To clarify - in first video there are frames like this:
where the flutter scroll view doesn't keep up with UITableView (on the right) because is always one frame behind. The video on the right stays aligned at all times.
knopp commentedon Aug 28, 2022
cc @iskakaushik
theniceboy commentedon Sep 1, 2022
https://www.youtube.com/watch?v=N66MdEDHh2Q
This is a slo-mo video recorded that can demonstrate this issue.
Up top is the Wonderous app, and down below is a swift app.
pedromassango commentedon Sep 5, 2022
@knopp were you able to reproduce this on Android? Seems iOS specific, can't reproduce on Android Emulator so far.
MobileMon commentedon Sep 7, 2022
jmagman commentedon Sep 7, 2022
cc @cyanglaz
Hixie commentedon Oct 4, 2022
@knopp btw please don't hesitate to land your test app in https://github.com/flutter/platform_tests if you want to. Collecting test apps like this is what that repo was made for.
cyanglaz commentedon Oct 25, 2022
I'm not currently working on this so I'll un-assign myself. I might circle back to this eventually.
@knopp Do you mind to open a draft PR with your engine modification that results the right video?
alid-code-archives commentedon Oct 25, 2022
@cyanglaz did you see this as being reasonably in-reach to resolve?
There's comments about Flutter being in an "uncanny valley" - close to native, yet slightly different in an uncanny way. Outside of just saying "performance" or "jank" I couldn't identify why scrolling feels off in every flutter app I tried. So I find relief at finding issues like this since they identify why that's the case, and fixing these issues overtime make true native experience feel feasible! 🤞
If I may suggest, this feels like a higher priority issue as scrolling is a major interaction in any app.
18 remaining items
tomsp-01 commentedon Jun 28, 2024
Maybe helpful additional info: This issue seems to only apply to IOS running 60hz.
I'll look further into resolving this issue soon.
Some of our customers noticed this issue running our app on "60hz IPhones"
knopp commentedon Jun 30, 2024
Couldn't it be just that is much less obvious at 120Hz? As far as I can tell the frame cycle should be the same, the latency should be still present just shorter in absolute numbers.
delfme commentedon Mar 3, 2025
I can confirm knopp guess, at 120hz the issue is less evident but still there. I also suspect this might be contributing to some occasional stutters. @knopp if turning the dirty patch into a PR doesn't take too much time, that would be greatly appreciated.
chinmaygarde commentedon Apr 3, 2025
Thanks to @loic-sharma for highlighting this issue.
Some context on why this behavior is observed, potential remedies, and gotchas.
The frame scheduling mechanism in Flutter wants to ensure three things:
With this in mind, when the framework requests an animation frame, it asks the engine to schedule a callback that gets called when the engine thinks it has the best shot of meeting these goals.
In this case, the engine first delivers input events to the framework. Since not all input events generate a frame, the engine will only schedule a frame when the framework asks it to. But here, the framework does. The engine arms the vsync waiter. The waiter fires at the next frame frame boundary and the framework works to submit a new frame within that interval for presentation at the next boundary. And that is the cause of delay.
I think a fix for this would involve weakening the guarantees around one or more of the three things the current scheduler is trying to achieve. We can say that during interactions, the fair chance guarantee and the pipelining is suspended. Frame consistency doesn’t matter in those cases. But, the engine will need to be told that it is under the influence of a user interaction. Today, there is no such mechanism. We'll also need to figure out some mechanism to avoid rendering multiple frames within one frame interval. With the advent of 120 Hz display this issue becomes less apparent but still present.
Also, on older phones, we have had to add workarounds for inconsistent delivery of input events on iOS. These could also be adding latency in some scenarios. I don’t think it's doing that here because it would affect both captures in Matt's experiment. But I think we should revisit that now too.
knopp commentedon Apr 4, 2025
Few things here:
I'm wondering if the inconsistent delivery had a root cause that we missed while adding a really complicated workaround - for example running CADisplayLink on main thread changes how often core animation picks-up layer updates, it also seems to affect event processing. I have plans to revisit this and maybe find less complicated solution for the event delivery if there is one. I have iPhone Xs somewhere, just need to find it :)
On iOS the swapchain images are simply IOSurfaces that we manage ourself in
FlutterMetalLayer. Because it mirrors the interface ofCAMetalLayerit is less ideal for platform view overlay with regard of memory usage, ideally we'd want to use same approach as on macOS where platform view overlays are much cheaper.It's been a while, but IIRC the primary issue for the latency is that we simply wait too long for the first frame. When touch events arrive, framework process the events and requests frame. The touch events are processed at the very beginning of frame cycle. There is almost entire frame duration available, but we ignore that waiting for display-link tick. The solution would be to detect this and fire the first frame callback immediately, why subsequent frames will be all synchronized to display link). We do similar thing on macOS, because CVDisplayLink fires 1.5 frames before the target time which wastes entire frame cycle, so we synthesize the first tick in the vsync waiter.
This is very much on my radar still, but requires deeper look at all the pieces (including how we're using the display link and event resampling) and needs more time than I can currently allocate.
knopp commentedon Apr 4, 2025
A lot of stuff changed since the touch resampling workaround was implemented (including not using CAMetaLayer anymore and always running display link on main thread), plus several iOS updates, so it's worth looking into if we still need it.
MTtankkeo commentedon Aug 22, 2026
@knopp Since you’ve already spent quite a bit of time investigating this part of the iOS presentation pipeline in #191388, I’d really appreciate your thoughts on this approach.
You previously pointed out that changes around Core Animation commit timing can be risky, and you’re much more familiar with the invariants in this area than I am. I’ve tried to keep this change narrowly scoped to FlutterMetalLayer.
If, after looking through the PR #191543, you think this approach is reasonable, I’d be very happy to work on this together. In particular, I’m not very familiar with the existing test infrastructure for this path, so your help with reviewing the correctness assumptions, identifying the right tests, or contributing tests directly would be extremely helpful.
If you contribute code or tests, I’d be happy to preserve your contribution directly in the commit history. I’d prefer to treat this as a collaborative effort rather than just asking you for a one-way review.
knopp commentedon Aug 22, 2026
@MTtankkeo, CA commit is scheduled right after display link callback. It is very unlikely that a roundtrip to raster thread and command buffer scheduling would be completed before. I think by the time the scheduled handler executes the layer tree has been long committed.
When event tracking UIKit (or CA) will stubbornly delay any layer update after the initial commit until the next frame. In my experience flushing the transaction in this case did nothing most of the time. I think the choice by the CA is deliberate.
I have a work in progress PR that addresses this #191388 (the last two commits are relevant to this). It is still something I'm playing with, I think ideally I want get to the point where our layout/paint will happen before UIKit and then UIKit updates can run in parallel with our raster thread. But I'm not quite there yet.
knopp commentedon Aug 22, 2026
I just noticed that you PR actually references #191388. I think I might need to re-read the description, I'm not certain what issue it solves. I originally though it was meant to be alternative solution.
MTtankkeo commentedon Aug 22, 2026
Yes, it’s related to #191388, but I don’t intend this PR as an alternative implementation of it.
The specific issue I’m trying to address here is narrower: FlutterDrawable::present() can publish the IOSurface to FlutterMetalLayer before the Metal command buffer producing that surface has been scheduled.
#191388 is useful context because it explores the broader Core Animation presentation/commit timing problem, but this PR is specifically about the Metal scheduling → IOSurface presentation handoff.
The manual comparison includes a build with both changes applied because I wanted to compare the combined result against native behavior, while also keeping separate builds to verify the effect of each change independently.
knopp commentedon Aug 22, 2026
I commented on your PR. Unless I'm missing something we already present from a scheduled handler:
flutter/engine/src/flutter/impeller/renderer/backend/metal/surface_mtl.mm
Lines 304 to 310 in 4ebf37f
MTtankkeo commentedon Sep 15, 2026
@knopp I just saw #191444. I had already addressed the input and VSync side using a different approach in my PR, but it appears that we are trying to solve the same part of the latency problem.
To avoid duplicating the work, I’m considering following the direction of your PR and narrowing mine to the remaining presentation side, specifically the platform-view-compatible CAMetalLayer path and GPU blit.
Would that division make sense to you? If so, I can narrow my PR to the presentation side.