Skip to content

[web:multi-view] cannot scroll page when cursor lands on a FlutterView #139263

Closed

Description

@yjbanov

When embedding a FlutterView onto a custom element, e.g. in multi-view mode, the app may have two sources of scrolling:

  1. Flutter scrolls some content inside itself.
  2. The host page scrolls its content, and one piece of content may be a FlutterView.

Case 1 seems to work well. However, case 2 does not work when the mouse cursor or finger (i.e. pointer) is on top of the FlutterView. When this happens the web engine calls preventDefault, which prevents browser scrolling. Somehow the engine needs to know when not to call preventDefault, but that information lives in the framework.

Demo: in the sample below, see that when the pointer is on top of the blue area the page is able to scroll, but if the pointer moved to the yellow area the scrolling no longer happens.

Screen.Recording.2023-11-29.at.12.54.43.PM.mov

A couple of potential fixes:

  • Change PointerDataPacketCallback to return some value to the engine that provides information about whether the pointer should be forwarded to the platform for handling.
  • As part of rendering the frame, perhaps in the Scene graph, annotate regions with information about how certain pointer events should be handled.

My gut feeling tells me that the first option is better. For one, it is simpler and less invasive. But also, the logic that determines what to do with a pointer may depend on many things - kind, device, location, pressure - and the second option would have to pre-declare all this information even if a tiny subset of it is applicable.

This may require a collaboration between the framework and the web teams, but I'm assigning it to the framework team first to get guidance on what option would be the most idiomatic and work well across platforms.

Activity

  1. added
    frameworkflutter/packages/flutter repository. See also f: labels.
    engineflutter/engine related. See also e: labels.
    platform-webWeb applications specifically
    P1High-priority issues at the top of the work list
    triaged-webTriaged by Web platform team
    on Nov 29, 2023
  2. goderbauer commented on Nov 29, 2023

    @goderbauer
    Member

    Some thoughts about this:

    Change PointerDataPacketCallback to return some value to the engine that provides information about whether the pointer should be forwarded to the platform for handling.

    Based on a single pointer event (e.g. a pointer down) we cannot decide whether the framework will "handle" the event. For example, when we see a pointer down event delivered via the PointerDataPacketCallback this could be part of a tap gesture (which we want to handle in the framework) or it could end up being a scroll (which we may want to handle in the browser). We will only know for sure after we have seen a sequence of follow-up events and only at that point can we tell you whether the framework handled them or not (basically, once we know if any gesture handler claimed victory in the gesture arena). This information wouldn't be available synchronously when pointer events are delivered to the framework.

    As part of rendering the frame, perhaps in the Scene graph, annotate regions with information about how certain pointer events should be handled.

    I think this has the same problem, but in reverse: How are you on the engine-side going to predict that a pointer down event is going to be a scroll gesture (and needs to be handled directly without telling the framework) vs. its going to be a tap gesture that the framework needs to know about to activate a button?

    Somehow, the gesture arena of the framework needs to work together with the gesture recognition system of the browser. Not sure how we can archive this, though, and if browsers have any APIs for this?

    /cc @mdebbar

  3. flutter-triage-bot commented on Dec 1, 2023

    @flutter-triage-bot

    The triaged-web label is irrelevant if there is no team-web label or fyi-web label.

  4. goderbauer commented on Jan 4, 2024

    @goderbauer
    Member

    Earlier @yjbanov and I had a conversation about this and came up with two potential ideas:

    Idea: Competition

    We send a specially crafted member representing the browser into the gesture arena for every pointer. It can only win if it is the last remaining member and no other member wins. It remembers all the pointer events dispatched to it and the platform prevents defaults for all of them. When this special member "wins" the arena, we play the events back to the platform and let it handle it since nobody on the framework side wants them.

    This will require some API on the platforms to inject events back into the platform. Such API may not exist.

    Idea: Cooperation

    For scroll wheel events we could check whether the PointerSignalResolver has any registered event handlers when PointerSignalResolver.resolve is called. If it has a registered handler for the event, then the event is handled by the framework and the platform is told synchronously to prevent defaults. If it has no registered handler, the platform needs to handle the event. This should basically resolve the problem altogether for scroll wheels because Scrollables only register a handler with the PointerSignalResolver if they can actually scroll in response to the event.

    For scroll by touch we could extend GestureArenaMember with an option to prevent defaults when a particular member enters the gesture arena on a point down event. By default, all GestureDetectors would allow defaults on the GestureArenaMember that they send to the arena - except for the member added by a vertically scrolling Scrollable. Pointer events are now dispatched as usual, but when at the end of dispatching a pointer down event the arena is closed we check if any members disallow defaults. If that is the case, the platform is told to prevent defaults, otherwise it isn't. This isn't expected to solve the issue in all cases, though: If a scrollable is all the way at the top it can still scroll down and would therefore send a default-preventing member into the arena on pointer down to prevent defaults even when the pointer down event would be the start of a scroll up gesture that should be handled by the platform, but wouldn't be in this scenario. An argument could be made here that UIs with nested scrolling are kinda bad and should be avoided. Either you want the FlutterView to be scrollable or the outer page, but not both at the same time. Therefore, this could be a "good enough" solution that fulfills most use cases.

    Both of these rely on the fact that upon receiving either a signal or a pointer down event from the platform we can synchronously message back to the platform whether defaults need to be prevented. In the framework, event handling is pretty much synchronous - with two known exceptions: locked events and resampling.

    When events are locked incoming pointer events are queued and processed after events are unlocked again. Currently, we only lock events while producing a warmup frame and during hot reload. Hot reload isn't available yet on the web, so we may not have to worry too much about it. And even if it were, it would probably be fine if events arriving during this dev operation behave a little strange - after all, the app is reassembling itself. The same can be said about the warm-up frame right at app startup (and after hot reload).

    If resampling is enabled (it is off by default) we may be out of luck since the whole point of it is to process event asynchronously. PointerSignalEvents appear to not be affected by resampling, so those could still work. It will need some additional investigation to see if PointerDown events are also dispatched asynchronously as part of resampling.

    There may be further asynchronicity in event dispatch on the engine side. At least when semantics are enabled, the web engine seems to operate in async mode for event dispatch (see here). Further investigation is needed to figure out if we could make that work with this approach.

    There are also the other non-web platforms to consider (supporting multi-view in an add-to-app fashion will surface the same problems there). They will likely involve a thread hop for event dispatch and further investigation will be needed to see if defaults can be prevented asynchronously for incoming pointer events.

  5. goderbauer commented on Jan 4, 2024

    @goderbauer
    Member

    Idea "competition" is kinda of how PlatformViews work today on at least iOS and (in the future) MacOS. It could likely also be used for view embedding on those platforms. (cc @cbracken)

    @yjbanov How does the web do gesture disambiguation with platform views there to decide whether the platform view or the surrounding flutter app should handle it?

  6. mdebbar commented on Jan 16, 2024

    @mdebbar
    Contributor

    @goderbauer thanks for the write up!

    When this special member "wins" the arena, we play the events back to the platform and let it handle it since nobody on the framework side wants them.

    AFAIK, this is not possible on the web. We can create an event and dispatch it to the DOM, but the browser won't handle it.

    For scroll by touch we could extend GestureArenaMember with an option to prevent defaults when a particular member enters the gesture arena on a point down event.

    Note: after playing a bit with this in a jsfiddle, I found out the event that needs to be preventDefaulted is touchmove with {passive: false}. Preventing default on pointermove, for example, won't work.

    How does the web do gesture disambiguation with platform views there to decide whether the platform view or the surrounding flutter app should handle it?

    It is my understanding that on the web, platform views always swallow pointer events. There are a few reasons:

    • In some cases, the browser doesn't even let us receive the events (e.g. if the platform view is an iframe).
    • We can't replay events later if we decide to let the platform view handle them.

    The solution we have right now is to let the app developer decide. If they want to let the platform view handle the events, then they don't have to do anything (this is the default behavior). But if they want to handle the events on the Flutter side, they can use pointer_interceptor.

  7. yjbanov commented on Jan 17, 2024

    @yjbanov
    ContributorAuthor

    Note: after playing a bit with this in a jsfiddle, I found out the event that needs to be preventDefaulted is touchmove with {passive: false}. Preventing default on pointermove, for example, won't work.

    A-ha! That's why it didn't work in my experiments. Thanks for digging this up!

  8. removed
    P1High-priority issues at the top of the work list
    on Feb 22, 2024
  9. 14 remaining items

  10. ditman commented on May 21, 2024

    @ditman
    Member

    @Garoth full-screening the Flutter view is a cool feature!

  11. ditman commented on Jun 29, 2024

    @ditman
    Member

    @Garoth looking at your code, I think this PR may be relevant to you:

  12. added
    P1High-priority issues at the top of the work list
    team-webOwned by Web platform team
    triaged-webTriaged by Web platform team
    and removed
    P2Important issues not at the top of the work list
    on Jul 17, 2024
  13. yjbanov commented on Jul 17, 2024

    @yjbanov
    ContributorAuthor

    Scrolling using the mouse and the touchpad has landed, which was in the MVP of the feature. Touch-based scrolling can be addressed as a follow-up.

  14. github-actions commented on Aug 1, 2024

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  15. locked as resolved and limited conversation to collaborators on Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listengineflutter/engine related. See also e: labels.frameworkflutter/packages/flutter repository. See also f: labels.platform-webWeb applications specificallyteam-webOwned by Web platform teamtriaged-webTriaged by Web platform team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions