Skip to content

Migrate rendering plugins to SurfaceProducer after stable release #139702

Closed
flutter/packages
#6989

Description

@matanlurey

tl;dr:

  • It is both non-trivial to support the Android SurfaceTexture API in Impeller (on Android with Vulkan)
  • SurfaceTexture is not a desirable integration point (based on talking to the Android team directly)
  • Usage is quite low, and there is an easy/mechanical path to migrate it to something better

Background

The Flutter engine on Android supports two external rendering sources, SurfaceTexture123 (an OpenGLES texture) and ImageReader456 (a GPU-ready buffer). Image.getHardwareBuffer, which is used by our ImageReader-based code path, is only supported on newer Android API versions (API level 28).

In Impeller, for the first time, we'll support Vulkan, starting at Android API version 29, over OpenGLES (all Flutter Android apps currently use Skia, which in turn uses OpenGLES). Unfortunately, it's non-trivial to use an OpenGLES texture (SurfaceTexture) in Vulkan, and after some internal discussion with the Android team, we've come to the conclusion that SurfaceTexture is not the right integration point in general.

I did a quick scan of usage of (create|register)SurfaceTexture, both internally and externally:

  • Internal numbers are small (single digits)
  • Within org: flutter, we have about 17 usages
  • Github-wide, I see <400 usages, many of which are ancient projects, forks, or otherwise not critical ecosystem pieces

Proposal

We'll add a new, parallel API/extension point, i.e. (create|register)Surface7. Similar to our platform view mechanics, we'll transparently either use SurfaceTexture (on older devices) or ImageReader (on newer devices), and end-users (users that consume plugins that use (create|register)Surface will not have to do or configure anything.

Timeline

Testing Strategy

Our ability to run comprehensive Android device tests in the engine repository is very limited (as of 2023-12-06):

  • Android engine unit tests are mock-heavy
  • Separation of framework and engine means to properly write an integration test you need code on "both" sides

@johnmccutchan coincidentally has a napkin-math proposal to improve the state of Android platform testing, which is critical for stability of our Android platform views and plugins. As necessary I plan to pair with him to iron out that proposal, land it, and then consume whatever infrastructure we built out/refactor out to also test our (create|register)Surface API.

Some scenarios will include:

  • An older Android {device, emulator} that will implicitly use surface textures
  • A newer Android {device, emulator} that will implicitly use hardware buffers (via ImageReader)
  • Try dynamically resizing the texture at runtime
  • Try "animated" textures (i.e. each frame is a different image)
  • Try tearing down and re-setting up views that use external textures

/cc @dnfield @jonahwilliams @gaaclarke @johnmccutchan @reidbaker

Footnotes

  1. android.graphics.SurfaceTexture ↩

  2. surface_texture_external_texture.h ↩

  3. platform_view_android.cc ↩

  4. android.media.ImageReader ↩

  5. image_external_texture.h ↩

  6. platform_view_android.cc ↩

  7. android.view.Surface ↩

Activity

  1. added
    platform-androidAndroid applications specifically
    engineflutter/engine related. See also e: labels.
    a: platform-viewsEmbedding Android/iOS views in Flutter apps
    c: proposalA detailed proposal for a change to Flutter
    P2Important issues not at the top of the work list
    c: tech-debtTechnical debt, code quality, testing, etc.
    e: impellerImpeller rendering backend issues and features requests
    fyi-androidFor the attention of Android platform team
    fyi-ecosystemFor the attention of Ecosystem team
    on Dec 6, 2023
  2. self-assigned this
    on Dec 6, 2023
  3. moved this to 🤔 Needs Triage in Impelleron Dec 6, 2023
  4. jonahwilliams commented on Dec 7, 2023

    @jonahwilliams
    Contributor

    FYI @stuartmorgan

    I think this is a reasonable proposal, the only caveat I would consider is how exactly we do the migration for the flutter owned packages and plugins. I think we have a generic policy of trying to support the current and last stable Flutter verisons, which on the surface (haha) would prevent us from starting the migration until after the next stable. Though Its possible that there is some compile time trickery and/or reflection we could do in Java to be backwards compatible.

  5. stuartmorgan-g commented on Dec 7, 2023

    @stuartmorgan-g
    Contributor
    • 5. Deprecate (create|register)SurfaceTexture. In Impeller, issue a warning at runtime if it is used.
    • 6. In Impeller, issue an error if it is used
    • 7. By the time Skia-Ganesh backend is removed, remove (create|register)SurfaceTexture entirely

    What is the proposed timeframe between step 5 and steps 6/7? Breaking plugins in the ecosystem by removing or breaking plugin-facing APIs is quite disruptive.

    the only caveat I would consider is how exactly we do the migration for the flutter owned packages and plugins. I think we have a generic policy of trying to support the current and last stable Flutter verisons, which on the surface (haha) would prevent us from starting the migration until after the next stable.

    Why would waiting until after the next stable to migrate our plugins be an issue? We should not be breaking the entire ecosystem on a shorter timeframe than what we would need for our own plugins regardless.

  6. 19 remaining items

  7. changed the title [-]Support external textures in Android+Impeller (replace `(create|register)SurfaceTexture`)[/-] [+]Migrate rendering plugins to `SurfaceProducer` after stable release[/+] on Feb 14, 2024
  8. jonahwilliams commented on May 29, 2024

    @jonahwilliams
    Contributor

    All plugin changes for the Flutter repo have landed, closing.

  9. github-actions commented on Jun 12, 2024

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  10. locked as resolved and limited conversation to collaborators on Jun 12, 2024
  11. unlocked this conversation on Jul 3, 2024
  12. matanlurey commented on Aug 23, 2024

    @matanlurey
    ContributorAuthor

    Update: @camsim99 landed flutter/packages#6856 (camerax) and flutter/packages#6989 is in the merge queue (video_player), which will, once merged, complete this issue out.

  13. github-actions commented on Sep 7, 2024

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  14. locked as resolved and limited conversation to collaborators on Sep 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Important issues not at the top of the work lista: platform-viewsEmbedding Android/iOS views in Flutter appsc: proposalA detailed proposal for a change to Flutterc: tech-debtTechnical debt, code quality, testing, etc.e: impellerImpeller rendering backend issues and features requestsengineflutter/engine related. See also e: labels.fyi-androidFor the attention of Android platform teamfyi-ecosystemFor the attention of Ecosystem teamplatform-androidAndroid applications specificallyteam-engineOwned by Engine teamtriaged-engineTriaged by Engine team

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions