Repository navigation
Description
tl;dr:
- It is both non-trivial to support the Android
SurfaceTextureAPI in Impeller (on Android with Vulkan) SurfaceTextureis not a desirable integration point (based on talking to the Android team directly)- Usage is quite low, and there is an easy/mechanical path to migrate it to something better
Background
The Flutter engine on Android supports two external rendering sources, SurfaceTexture123 (an OpenGLES texture) and ImageReader456 (a GPU-ready buffer). Image.getHardwareBuffer, which is used by our ImageReader-based code path, is only supported on newer Android API versions (API level 28).
In Impeller, for the first time, we'll support Vulkan, starting at Android API version 29, over OpenGLES (all Flutter Android apps currently use Skia, which in turn uses OpenGLES). Unfortunately, it's non-trivial to use an OpenGLES texture (SurfaceTexture) in Vulkan, and after some internal discussion with the Android team, we've come to the conclusion that SurfaceTexture is not the right integration point in general.
I did a quick scan of usage of (create|register)SurfaceTexture, both internally and externally:
- Internal numbers are small (single digits)
- Within
org: flutter, we have about 17 usages - Github-wide, I see <400 usages, many of which are ancient projects, forks, or otherwise not critical ecosystem pieces
Proposal
We'll add a new, parallel API/extension point, i.e. (create|register)Surface7. Similar to our platform view mechanics, we'll transparently either use SurfaceTexture (on older devices) or ImageReader (on newer devices), and end-users (users that consume plugins that use (create|register)Surface will not have to do or configure anything.
Timeline
- 1. Add a new API,
(create|register)Surface, which delegates (internally) toSurfaceTextureorImageReader - 2. Migrate internal plugins
- 3. Migrate
org: flutterplugins and examples - 4. Add user-facing documentation on flutter.dev about the recommended approach
- 5. Deprecate
(create|register)SurfaceTexture. In Impeller, issue a warning at runtime if it is used (Proposal: Decide what parts of the Java embedder rendering APIs to deprecate #148557) - 6. In Impeller, issue an error if it is used (Done: Issue an
ERRORinstead of anINFOfor a non-working API. flutter-team-archive/engine#52892) - 7. By the time Skia-Ganesh backend is removed, remove
(create|register)SurfaceTextureentirely
Testing Strategy
Our ability to run comprehensive Android device tests in the engine repository is very limited (as of 2023-12-06):
- Android engine unit tests are mock-heavy
- Separation of framework and engine means to properly write an integration test you need code on "both" sides
@johnmccutchan coincidentally has a napkin-math proposal to improve the state of Android platform testing, which is critical for stability of our Android platform views and plugins. As necessary I plan to pair with him to iron out that proposal, land it, and then consume whatever infrastructure we built out/refactor out to also test our (create|register)Surface API.
Some scenarios will include:
- An older Android {device, emulator} that will implicitly use surface textures
- A newer Android {device, emulator} that will implicitly use hardware buffers (via
ImageReader) - Try dynamically resizing the texture at runtime
- Try "animated" textures (i.e. each frame is a different image)
- Try tearing down and re-setting up views that use external textures
/cc @dnfield @jonahwilliams @gaaclarke @johnmccutchan @reidbaker
Footnotes
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fields🤔 Needs Triage
Activity
jonahwilliams commentedon Dec 7, 2023
FYI @stuartmorgan
I think this is a reasonable proposal, the only caveat I would consider is how exactly we do the migration for the flutter owned packages and plugins. I think we have a generic policy of trying to support the current and last stable Flutter verisons, which on the surface (haha) would prevent us from starting the migration until after the next stable. Though Its possible that there is some compile time trickery and/or reflection we could do in Java to be backwards compatible.
stuartmorgan-g commentedon Dec 7, 2023
What is the proposed timeframe between step 5 and steps 6/7? Breaking plugins in the ecosystem by removing or breaking plugin-facing APIs is quite disruptive.
Why would waiting until after the next stable to migrate our plugins be an issue? We should not be breaking the entire ecosystem on a shorter timeframe than what we would need for our own plugins regardless.
19 remaining items
[-]Support external textures in Android+Impeller (replace `(create|register)SurfaceTexture`)[/-][+]Migrate rendering plugins to `SurfaceProducer` after stable release[/+]Migrate Android `scenario_app` to the `SurfaceProducer` API (#50993)
Reverts "Migrate Android `scenario_app` to the `SurfaceProducer` API (#…
Issue an`ERROR` instead of an `INFO` for a non-working API. (#52892)
jonahwilliams commentedon May 29, 2024
All plugin changes for the Flutter repo have landed, closing.
github-actions commentedon Jun 12, 2024
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.matanlurey commentedon Aug 23, 2024
Update: @camsim99 landed flutter/packages#6856 (
camerax) and flutter/packages#6989 is in the merge queue (video_player), which will, once merged, complete this issue out.github-actions commentedon Sep 7, 2024
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.