Skip to content

[Impeller] Support external (surface) textures in Vulkan #137639

Closed

Description

@matanlurey

Note

This is a quasi-placeholder issue, as I only learned what a "Surface Texture" is ~5 minutes ago, so part of the resolution will be updating this issue with actual useful information. Thanks for your patience 😄


An Android SurfaceTexture is a combination of a surface and an OpenGL ES (GLES) texture 1.

For example, this would be used to display frames from a camera plugin:

On newer devices (or is it APIs? I'm not sure), Android provides a backend-agnostic hardware buffer that can be used (directly) by Vulkan but on older devices we'll receive an OpenGL-backed surface texture, and need to adapt it for use within Vulkan - that will involve creating a new OpenGL context, and some interop.

Some possible references:

As an initial look, I'll try package:camera in an Impeller+Vulkan demo. It might also be worth looking into Skia's codebase to see how they handle this.

/cc @jonahwilliams @johnmccutchan (feel free to edit this description if inaccurate!)

Footnotes

  1. https://source.android.com/docs/core/graphics/arch-st ↩

Activity

  1. added
    engineflutter/engine related. See also e: labels.
    P1High-priority issues at the top of the work list
    c: parityWorks on one platform but not another
    e: impellerImpeller rendering backend issues and features requests
    fyi-androidFor the attention of Android platform team
    on Oct 31, 2023
  2. self-assigned this
    on Oct 31, 2023
  3. moved this to 🤔 Needs Triage in Impelleron Oct 31, 2023
  4. moved this from 🤔 Needs Triage to 🖖 Vulkan in Impelleron Oct 31, 2023
  5. jonahwilliams commented on Oct 31, 2023

    @jonahwilliams
    Contributor

    On newer devices (or is it APIs? I'm not sure), Android provides a backend-agnostic hardware buffer that can be used (directly) by Vulkan but on older devices we'll receive an OpenGL-backed surface texture, and need to adapt it for use within Vulkan - that will involve creating a new OpenGL context, and some interop.

    Individual APIs and packages may still provide surface textures even on devices that support hardware buffers.

    The Flutter Android engine also has public APIs for registering external textures via SurfaceTexture.

  6. johnmccutchan commented on Oct 31, 2023

    @johnmccutchan
    Contributor

    Jonah is correct, SurfaceTexture support is part of our public API and so any application is free to give us a SurfaceTexture and ask Flutter to render it as a texture layer forever regardless of Android version.

    Based on research I did, a tractable path to solving this problem is to:

    1. Allocate a Vulkan texture
    2. Use the GL external memory extensions to create a rendering target that is backed by the Vulkan allocated texture.
    3. Do a blit pass in GL to copy the GL_TEXTURE_EXTERNAL_OES texture into the Vulkan image.

    I've also dug into the implementation of SurfaceTextures and internally they are backed by HardwareBuffers, there is even a non-public NDK function that will retrieve the HardwareBuffer from the SurfaceTexture. I was planning on proposing that Android make the function public in a future SDK release. We would still need the above approach in the meantime.

  7. matanlurey commented on Oct 31, 2023

    @matanlurey
    ContributorAuthor
  8. matanlurey commented on Oct 31, 2023

    @matanlurey
    ContributorAuthor

    I have a basic app that draws a yellow texture from the Android side to render on the Dart side.

    Kotlin
    package com.example.surface_texture
    
    import android.graphics.SurfaceTexture
    import android.view.Surface
    import androidx.annotation.NonNull
    import io.flutter.embedding.engine.plugins.FlutterPlugin
    import io.flutter.plugin.common.MethodCall
    import io.flutter.plugin.common.MethodChannel
    import io.flutter.plugin.common.MethodChannel.MethodCallHandler
    
    class SurfaceTexturePlugin : FlutterPlugin, MethodCallHandler {
        private lateinit var channel: MethodChannel
        private lateinit var mFlutterPluginBinding: FlutterPlugin.FlutterPluginBinding
        private lateinit var mSurfaceTexture: SurfaceTexture
        private lateinit var mSurface: Surface
    
        override fun onAttachedToEngine(@NonNull binding: FlutterPlugin.FlutterPluginBinding) {
            mFlutterPluginBinding = binding
            channel = MethodChannel(mFlutterPluginBinding.binaryMessenger, "surface_texture")
            channel.setMethodCallHandler(this)
        }
    
        override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: MethodChannel.Result) {
            when (call.method) {
                "generateSurfaceTexture" -> {
                    val surfaceHeight = call.argument<Int>("height")
                    val textureId = generateSurfaceTexture(surfaceHeight ?: 0)
                    result.success(textureId)
                }
                "setSurfaceBufferSize" -> {
                    val surfaceHeight = call.argument<Int>("height");
                    setSurfaceBufferSize(surfaceHeight ?: 0)
                    result.success(null)
                }
                else -> {
                    result.notImplemented()
                }
            }
        }
    
        override fun onDetachedFromEngine(binding: FlutterPlugin.FlutterPluginBinding) {
            channel.setMethodCallHandler(null)
        }
    
        private fun generateSurfaceTexture(surfaceHeight: Int): Long {
            val textureRegistry = mFlutterPluginBinding.textureRegistry
            val surfaceTextureEntry = textureRegistry.createSurfaceTexture()
            mSurfaceTexture = surfaceTextureEntry.surfaceTexture()
            mSurfaceTexture.setDefaultBufferSize(1080, surfaceHeight)
            mSurface = Surface(mSurfaceTexture)
    
            val canvas = mSurface.lockCanvas(null)
            canvas.drawRGB(255, 230, 15)
            mSurface.unlockCanvasAndPost(canvas)
    
            return surfaceTextureEntry.id()
        }
    
        private fun setSurfaceBufferSize(surfaceHeight: Int) {
            mSurfaceTexture.setDefaultBufferSize(1080,surfaceHeight)
            val canvas = mSurface.lockCanvas(null)
            canvas.drawRGB(255, 230, 15)
            mSurface.unlockCanvasAndPost(canvas)
        }
    }
    Dart
    import 'package:flutter/material.dart';
    import 'package:flutter/services.dart';
    
    void main() {
      runApp(const MainApp());
    }
    
    final class SurfaceTexturePlugin {
      static const _channel = MethodChannel('surface_texture');
    
      static Future<int?> generateSurfaceTexture({int height = 0}) async {
        return await _channel.invokeMethod(
          'generateSurfaceTexture',
          {'height': height},
        );
      }
    
      static Future<void> setSurfaceBufferSize({int height = 0}) async {
        return await _channel.invokeMethod(
          'setSurfaceBufferSize',
          {'height': height},
        );
      }
    }
    
    final class MainApp extends StatefulWidget {
      const MainApp({super.key});
    
      @override
      State<MainApp> createState() => _MainAppState();
    }
    
    final class _MainAppState extends State<MainApp> {
      int? _textureId;
      double _height = 0;
    
      @override
      void initState() {
        super.initState();
        _initTextureState();
      }
    
      Future<void> _initTextureState() async {
        int? textureId;
    
        try {
          textureId = await SurfaceTexturePlugin.generateSurfaceTexture(
            height: _height.toInt(),
          );
        } on PlatformException {
          debugPrint('Failed to generate surface texture.');
        }
    
        if (!mounted) {
          return;
        }
    
        setState(() {
          _textureId = textureId;
        });
      }
    
      @override
      Widget build(BuildContext context) {
        return MaterialApp(
          home: Scaffold(
              appBar: AppBar(
                title: const Text('Surface Texture Demo'),
              ),
              body: _textureId == null
                  ? const Center(child: CircularProgressIndicator())
                  : SingleChildScrollView(
                      child: Column(
                        children: [
                          Text('Texture ID: $_textureId, Height: $_height'),
                          Slider(
                            value: _height,
                            min: 0,
                            max: 1080 * 2,
                            onChanged: (value) {
                              setState(() {
                                _height = value;
                              });
                              SurfaceTexturePlugin.setSurfaceBufferSize(
                                height: _height.toInt(),
                              );
                            },
                          ),
                          _Texture(textureId: _textureId!, height: _height),
                        ],
                      ),
                    )),
        );
      }
    }
    
    final class _Texture extends StatelessWidget {
      final int _textureId;
      final double _surfaceHeight;
    
      const _Texture({
        required int textureId,
        double height = 0,
      })  : _textureId = textureId,
            _surfaceHeight = height;
    
      @override
      Widget build(BuildContext context) {
        return Center(
          child: SizedBox(
            width: 1080,
            height: _surfaceHeight,
            child: Texture(textureId: _textureId),
          ),
        );
      }
    }

    With the Skia backend, it works as expected:

    With Impeller (Vulkan, Pixel 5):

  9. matanlurey commented on Nov 1, 2023

    @matanlurey
    ContributorAuthor

    For my own self discovery, how this works from framework <-> engine is:

    https://github.com/flutter/engine/blob/406f0c477afb8b57caa24bd1040af5596815c2c6/flow/layers/texture_layer.cc#L50-L69

  10. matanlurey commented on Nov 1, 2023

    @matanlurey
    ContributorAuthor

    flutter-team-archive/engine#36498 added support for iOS (though I think this PR meant to say, "on iOS in Impeller"). Also a quick check, if I try on an OpenGL device (Moto G4), I get a hard crash (where on a Vulkan device, nothing renders but there is no crash):

    Abort message: '[FATAL:flutter/impeller/renderer/backend/gles/proc_table_gles.h(38)] Fatal GL Error GL_INVALID_ENUM(1280) encountered on call to glTexParameteri
  11. matanlurey commented on Nov 1, 2023

    @matanlurey
    ContributorAuthor

    @jonahwilliams Can you clarify if the goal is to only wire up Vulkan and did you expect GLES to be working already?

  12. jonahwilliams commented on Nov 1, 2023

    @jonahwilliams
    Contributor

    GLES is a separate issue

  13. matanlurey commented on Nov 1, 2023

    @matanlurey
    ContributorAuthor

    Chatted with @jonahwilliams:

    For what needs to be done with Impeller I would look at John's more recent PRs.

    Some examples:

    That does most of the plumbing on GLES. But the external/is_wrapped flag isn't something that will be usable - that only works for GLES - GLES interop, not GLES Vulkan interop

    You'll want to look at surface_texture_external_texture variants

    See surface_texture_external_texture.h and surface_texture_external_texture_gl.h.

    that is where you will need to 1) allocate a vulkan texture with the extensions/external info that John linked in the bug and 2) copy the OES texture into that vulkan texture

  14. matanlurey commented on Nov 1, 2023

    @matanlurey
    ContributorAuthor

    @jonahwilliams Just to clarify before I get started (I realize it will take a few iterations), my goal is to do something similar to flutter-team-archive/engine#44734 (comment), i.e. I'd end up with a surface_texture_external_texture_vk.[cc|h], and obviously have it working within the Vulkan backend as well :)

  15. jonahwilliams commented on Nov 1, 2023

    @jonahwilliams
    Contributor

    Yeah that seems like a good place to start.

  16. matanlurey commented on Nov 7, 2023

    @matanlurey
    ContributorAuthor

    I made some progress in flutter-team-archive/engine#47632 on being able to test the texture selection logic, but ran into too many hickups (#138015, for example), and found out that it doesn't appear possible to setup a (real) EGL surface in the Android emulator (or it was not trivial):

    $ adb logcat
    
    11-07 12:56:58.180 10067 10067 E flutter : [ERROR:flutter/shell/platform/android/platform_view_android.cc(75)] CreateAndroidContext
    11-07 12:56:58.263 10067 10067 D libEGL  : loaded /vendor/lib64/egl/libEGL_emulation.so
    11-07 12:56:58.268 10067 10067 D libEGL  : loaded /vendor/lib64/egl/libGLESv1_CM_emulation.so
    11-07 12:56:58.271 10067 10067 D libEGL  : loaded /vendor/lib64/egl/libGLESv2_emulation.so
    11-07 12:56:58.308 10067 10067 E flutter : [ERROR:flutter/shell/platform/android/android_context_gl_skia.cc(87)] Could not choose an EGL configuration.
    11-07 12:56:58.308 10067 10067 E flutter : [ERROR:flutter/shell/platform/android/android_egl_surface.cc(52)] EGL Error: EGL_SUCCESS (12288)
    11-07 12:56:58.309 10067 10067 F flutter : [FATAL:flutter/shell/platform/android/platform_view_android.cc(153)] Check failed: android_context_->IsValid(). Could not create surface from invalid Android context.

    Given this isn't making any real progress towards getting surface textures working, I'm going to halt trying to have a good testing workflow (outside of manual tests) until I have something working end-to-end locally, and then explore some refactors (like #137798) to make testing the code easier.

  17. moved this from 🖖 Vulkan to ⚙️ In Progress in Impelleron Dec 2, 2023
  18. matanlurey commented on Dec 6, 2023

    @matanlurey
    ContributorAuthor

    Closing this issue in favor of #139702. Thanks!

  19. moved this from ⚙️ In Progress to ✅ Done in Impelleron Dec 6, 2023
  20. github-actions commented on Dec 26, 2023

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  21. locked as resolved and limited conversation to collaborators on Dec 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listc: parityWorks on one platform but not anothere: impellerImpeller rendering backend issues and features requestsengineflutter/engine related. See also e: labels.fyi-androidFor the attention of Android platform teamteam-engineOwned by Engine teamtriaged-engineTriaged by Engine team

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions