Skip to content

Flutter adds additional one frame touch latency compared to native #110431

Open

Description

@knopp
Bad Expected
tracking-bad.mov
tracking-expected.mov

Repository: https://github.com/knopp/flutter_scroll_overlay

When touch event is received, instead of immediately beginning a frame Flutter waits for next vsync. But that skips an entire frame. By the time vsync comes, the frame should be already rendered and ready to go. But instead it has just began. On iOS touch events seem to be delivered within few milliseconds of the display link callback so there should be enough time.

You can pause the "bad" video to see that during scrolling Flutter lags behind native by one frame exactly.

The expected video is with quick and dirty engine modification that forces BeginFrame immediately on input event.

The way vsync is implemented in Flutter might need some work. It should make sure that repaints are synchronized with display updates, but it shouldn't be causing additional latency.

Activity

  1. knopp commented on Aug 28, 2022

    @knopp
    MemberAuthor

    To clarify - in first video there are frames like this:

    Screenshot 2022-08-28 at 14 16 13

    where the flutter scroll view doesn't keep up with UITableView (on the right) because is always one frame behind. The video on the right stays aligned at all times.

  2. added
    engineflutter/engine related. See also e: labels.
    a: fidelityMatching the OEM platforms better
    on Aug 28, 2022
  3. knopp commented on Aug 28, 2022

    @knopp
    MemberAuthor
  4. theniceboy commented on Sep 1, 2022

    @theniceboy
    Contributor

    https://www.youtube.com/watch?v=N66MdEDHh2Q

    This is a slo-mo video recorded that can demonstrate this issue.

    Up top is the Wonderous app, and down below is a swift app.

  5. pedromassango commented on Sep 5, 2022

    @pedromassango
    Member

    @knopp were you able to reproduce this on Android? Seems iOS specific, can't reproduce on Android Emulator so far.

  6. added
    P2Important issues not at the top of the work list
    on Sep 6, 2022
  7. MobileMon commented on Sep 7, 2022

    @MobileMon
  8. jmagman commented on Sep 7, 2022

    @jmagman
    Member
  9. self-assigned this
    on Sep 7, 2022
  10. Hixie commented on Oct 4, 2022

    @Hixie
    Contributor

    @knopp btw please don't hesitate to land your test app in https://github.com/flutter/platform_tests if you want to. Collecting test apps like this is what that repo was made for.

  11. cyanglaz commented on Oct 25, 2022

    @cyanglaz

    I'm not currently working on this so I'll un-assign myself. I might circle back to this eventually.
    @knopp Do you mind to open a draft PR with your engine modification that results the right video?

  12. removed their assignment
    on Oct 25, 2022
  13. alid-code-archives commented on Oct 25, 2022

    @alid-code-archives

    @cyanglaz did you see this as being reasonably in-reach to resolve?

    There's comments about Flutter being in an "uncanny valley" - close to native, yet slightly different in an uncanny way. Outside of just saying "performance" or "jank" I couldn't identify why scrolling feels off in every flutter app I tried. So I find relief at finding issues like this since they identify why that's the case, and fixing these issues overtime make true native experience feel feasible! 🤞

    If I may suggest, this feels like a higher priority issue as scrolling is a major interaction in any app.

  14. 18 remaining items

  15. tomsp-01 commented on Jun 28, 2024

    @tomsp-01

    Maybe helpful additional info: This issue seems to only apply to IOS running 60hz.

    I'll look further into resolving this issue soon.

    Some of our customers noticed this issue running our app on "60hz IPhones"

  16. knopp commented on Jun 30, 2024

    @knopp
    MemberAuthor

    Couldn't it be just that is much less obvious at 120Hz? As far as I can tell the frame cycle should be the same, the latency should be still present just shorter in absolute numbers.

  17. delfme commented on Mar 3, 2025

    @delfme

    Maybe helpful additional info: This issue seems to only apply to IOS running 60hz.

    I'll look further into resolving this issue soon.

    Some of our customers noticed this issue running our app on "60hz IPhones"

    I can confirm knopp guess, at 120hz the issue is less evident but still there. I also suspect this might be contributing to some occasional stutters. @knopp if turning the dirty patch into a PR doesn't take too much time, that would be greatly appreciated.

  18. chinmaygarde commented on Apr 3, 2025

    @chinmaygarde
    Contributor

    Thanks to @loic-sharma for highlighting this issue.

    Some context on why this behavior is observed, potential remedies, and gotchas.

    The frame scheduling mechanism in Flutter wants to ensure three things:

    • Repaints are synchronized with display updates.
    • Each frame is given a “fair” chance to create its rendering intent without risking jank while ensuring consistent frame rates.
      • If one frame can take up to 16ms to render without causing stutter, the next frame cannot be constrained to render in some arbitrarily shorter amount of time.
      • If the sum of the UI and Raster thread workloads exceeds one frame interval, the next frame can be actively built while the previous one is being rendered. That is, their workloads must be able to be overlapped (on their own threads) so we can hit a consistent frame rate.
      • If the frame cannot consistently render in one frame interval, it must be given the leeway to render in some integer multiple of one frame interval. The notion is that it is better to operate consistently at 30Hz than it is to jump rapidly between 30Hz & 60Hz.
    • The renderer gets to render directly onto swapchain images.
      • The alternative would be to render into offscreen textures with those textures composited onscreen (synchronized with display updates). The offscreen textures would be massive (more so for wide-gamut rendering) and need to be maintained for each platform view interleaving level.
      • The constraint here is that trying to acquire a swapchain image too quickly will cause stalls. These will be enforced by the driver and will be idle CPU waits.

    With this in mind, when the framework requests an animation frame, it asks the engine to schedule a callback that gets called when the engine thinks it has the best shot of meeting these goals.

    In this case, the engine first delivers input events to the framework. Since not all input events generate a frame, the engine will only schedule a frame when the framework asks it to. But here, the framework does. The engine arms the vsync waiter. The waiter fires at the next frame frame boundary and the framework works to submit a new frame within that interval for presentation at the next boundary. And that is the cause of delay.

    I think a fix for this would involve weakening the guarantees around one or more of the three things the current scheduler is trying to achieve. We can say that during interactions, the fair chance guarantee and the pipelining is suspended. Frame consistency doesn’t matter in those cases. But, the engine will need to be told that it is under the influence of a user interaction. Today, there is no such mechanism. We'll also need to figure out some mechanism to avoid rendering multiple frames within one frame interval. With the advent of 120 Hz display this issue becomes less apparent but still present.

    Also, on older phones, we have had to add workarounds for inconsistent delivery of input events on iOS. These could also be adding latency in some scenarios. I don’t think it's doing that here because it would affect both captures in Matt's experiment. But I think we should revisit that now too.

  19. knopp commented on Apr 4, 2025

    @knopp
    MemberAuthor

    Few things here:

    1. I'm wondering if the inconsistent delivery had a root cause that we missed while adding a really complicated workaround - for example running CADisplayLink on main thread changes how often core animation picks-up layer updates, it also seems to affect event processing. I have plans to revisit this and maybe find less complicated solution for the event delivery if there is one. I have iPhone Xs somewhere, just need to find it :)

    2. On iOS the swapchain images are simply IOSurfaces that we manage ourself in FlutterMetalLayer. Because it mirrors the interface of CAMetalLayer it is less ideal for platform view overlay with regard of memory usage, ideally we'd want to use same approach as on macOS where platform view overlays are much cheaper.

    3. It's been a while, but IIRC the primary issue for the latency is that we simply wait too long for the first frame. When touch events arrive, framework process the events and requests frame. The touch events are processed at the very beginning of frame cycle. There is almost entire frame duration available, but we ignore that waiting for display-link tick. The solution would be to detect this and fire the first frame callback immediately, why subsequent frames will be all synchronized to display link). We do similar thing on macOS, because CVDisplayLink fires 1.5 frames before the target time which wastes entire frame cycle, so we synthesize the first tick in the vsync waiter.

    This is very much on my radar still, but requires deeper look at all the pieces (including how we're using the display link and event resampling) and needs more time than I can currently allocate.

  20. knopp commented on Apr 4, 2025

    @knopp
    MemberAuthor

    A lot of stuff changed since the touch resampling workaround was implemented (including not using CAMetaLayer anymore and always running display link on main thread), plus several iOS updates, so it's worth looking into if we still need it.

  21. self-assigned this
    on Jun 4, 2026
  22. MTtankkeo commented on Aug 22, 2026

    @MTtankkeo

    @knopp Since you’ve already spent quite a bit of time investigating this part of the iOS presentation pipeline in #191388, I’d really appreciate your thoughts on this approach.

    You previously pointed out that changes around Core Animation commit timing can be risky, and you’re much more familiar with the invariants in this area than I am. I’ve tried to keep this change narrowly scoped to FlutterMetalLayer.

    If, after looking through the PR #191543, you think this approach is reasonable, I’d be very happy to work on this together. In particular, I’m not very familiar with the existing test infrastructure for this path, so your help with reviewing the correctness assumptions, identifying the right tests, or contributing tests directly would be extremely helpful.

    If you contribute code or tests, I’d be happy to preserve your contribution directly in the commit history. I’d prefer to treat this as a collaborative effort rather than just asking you for a one-way review.

  23. knopp commented on Aug 22, 2026

    @knopp
    MemberAuthor

    @MTtankkeo, CA commit is scheduled right after display link callback. It is very unlikely that a roundtrip to raster thread and command buffer scheduling would be completed before. I think by the time the scheduled handler executes the layer tree has been long committed.

    When event tracking UIKit (or CA) will stubbornly delay any layer update after the initial commit until the next frame. In my experience flushing the transaction in this case did nothing most of the time. I think the choice by the CA is deliberate.

    I have a work in progress PR that addresses this #191388 (the last two commits are relevant to this). It is still something I'm playing with, I think ideally I want get to the point where our layout/paint will happen before UIKit and then UIKit updates can run in parallel with our raster thread. But I'm not quite there yet.

  24. knopp commented on Aug 22, 2026

    @knopp
    MemberAuthor

    I just noticed that you PR actually references #191388. I think I might need to re-read the description, I'm not certain what issue it solves. I originally though it was meant to be alternative solution.

  25. MTtankkeo commented on Aug 22, 2026

    @MTtankkeo

    I just noticed that you PR actually references #191388. I think I might need to re-read the description, I'm not certain what issue it solves. I originally though it was meant to be alternative solution.

    Yes, it’s related to #191388, but I don’t intend this PR as an alternative implementation of it.

    The specific issue I’m trying to address here is narrower: FlutterDrawable::present() can publish the IOSurface to FlutterMetalLayer before the Metal command buffer producing that surface has been scheduled.

    #191388 is useful context because it explores the broader Core Animation presentation/commit timing problem, but this PR is specifically about the Metal scheduling → IOSurface presentation handoff.

    The manual comparison includes a build with both changes applied because I wanted to compare the combined result against native behavior, while also keeping separate builds to verify the effect of each change independently.

  26. knopp commented on Aug 22, 2026

    @knopp
    MemberAuthor

    I commented on your PR. Unless I'm missing something we already present from a scheduled handler:

    // The drawable may come from a FlutterMetalLayer, so it can't be
    // presented through the command buffer.
    id<CAMetalDrawable> drawable = drawable_;
    [command_buffer addScheduledHandler:^(id<MTLCommandBuffer> buffer) {
    [drawable present];
    }];
    [command_buffer commit];

  27. MTtankkeo commented on Sep 15, 2026

    @MTtankkeo

    @knopp I just saw #191444. I had already addressed the input and VSync side using a different approach in my PR, but it appears that we are trying to solve the same part of the latency problem.

    To avoid duplicating the work, I’m considering following the direction of your PR and narrowing mine to the remaining presentation side, specifically the platform-view-compatible CAMetalLayer path and GPU blit.

    Would that division make sense to you? If so, I can narrow my PR to the presentation side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Important issues not at the top of the work lista: fidelityMatching the OEM platforms betterengineflutter/engine related. See also e: labels.platform-iosiOS applications specificallyteam-iosOwned by iOS platform teamtriaged-iosTriaged by iOS platform team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions